Received February 12, 2022, accepted February 17, 2022, date of publication February 21, 2022, date of current

version March 8, 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3153071

Optimal Grasping Strategy for Robots With a

Parallel Gripper Based on Feature Sensing
of 3D Object Model
HSIN-HAN CHIANG 1 , (Senior Member, IEEE), JIUN-KAI YOU2 ,
CHEN-CHIEN JAMES HSU 2 , (Senior Member, IEEE), AND JUN JO3
1 Department of Vehicle Engineering, National Taipei University of Technology, Taipei City 106, Taiwan
2 Department of Electrical Engineering, National Taiwan Normal University, Taipei City 106, Taiwan
3 School of Information and Communication Technology, Griffith University, Southport, QLD 4222, Australia
Corresponding author: Chen-Chien James Hsu ([email protected])
This work was supported in part by the Chinese Language and Technology Center of the National Taiwan Normal University (NTNU)
through the Featured Areas Research Center Program within the Framework of the Higher Education Sprout Project by the Ministry of
Education (MOE), Taiwan; and in part by the Ministry of Science and Technology, Taiwan, through the Pervasive Artificial Intelligence
Research (PAIR) Laboratories, under Grant MOST 109-2634-F-003-006 and Grant MOST 109-2634-F-003-007.

ABSTRACT Grasping strategy of many different kinds of target objects is vital in a wide range of automatic
robotic manipulations. Among the many potential applications for a robot arm with a parallel gripper, the
key challenge for robotic grasping is to determine an optimal grasping pose relative to the target object.
Previous works based on 2D grasp and 6-Dof grasp planning have been proposed to efficiently consider the
physical contact between the robotic gripper and the object. However, there are still a few unsolved problems
caused by partial and limited information about the detected objects due to their locations and geometries
that reduce the grasping quality and reliability. In view of these problems, this paper proposes an optimal
grasping strategy to deal with target objects with any poses based on their 3D model during the grasping
process. Experimental results of the performance evaluation show that the proposed method outperforms the
state-of-the-art in terms of grasping success rate on the YCB-Video datasets. Moreover, we further investigate
the effectiveness of the proposed method in two scenarios where the robotic manipulator works in either the
collaborative or bin-picking modes.

INDEX TERMS Robot grasping, point cloud, parallel gripper, 3D objects, grasp planning.

I. INTRODUCTION generally rely on professional engineers to customize control

With the advancement of science and technology in recent programs to achieve various grasping tasks on the production
years, robot arms have been widely used in many practical line for different applications. As a result, most applica-
fields, such as industrial manufacturing, living assistance, tions are based on low-mix, high-volume production models.
agriculture, medical rescue, entertainment services, military According to the study [2], production-grade robot systems
security, and even space exploration, etc. According to the can require days or weeks of effort from highly trained
International Federation of Robotics (IFR), over the past robot programmers when the production lines change. This
ten years, more than 2.7 million industrial robots have been adjustment time is far from satisfying the flexible produc-
introduced and operated worldwide [1]. The top three indus- tion needs of the industry, especially for small and medium-
tries with the highest usage are automobile (30%), electri- sized businesses (SMBs) that generally produce customized
cal and electronic industries (25%), and the metal industry products in small batches and short production cycles. There
(10%). Most of these traditional industrial robots operate is not sufficient capital or time to carry out sophisticated
automatically but lack self-awareness and flexibility. They tuning for robots in the production line. These difficulties
have prevented SMBs from adopting robot arms into produc-
The associate editor coordinating the review of this manuscript and tion lines to achieve Low Volume Automation (LVA). Thus,
approving it for publication was Bidyadhar Subudhi . robotic systems’ development has gradually shifted their

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
24056 VOLUME 10, 2022
H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

focus from efficient single repetitive tasks in a controllable in the accuracy of object pose estimation and the suitability
environment to intelligent robots with autonomous visual of the predefined grasp pose.
recognition and smart grasping strategy. As well as achieving As an attempt to relieve the restricted constraints in the
LVA in the factory, collaborative robots have also become aforementioned research, this paper presents a system that
increasingly popular in recent years. Through human- takes advantage of the related methods to deal with the above
robot interaction, a collaborative robot can work together mentioned problems so that the robotic manipulator can suc-
with people to accomplish various tasks, like assembling cessfully grasp the target object. Thanks to the rapid develop-
parts, carrying items, inspecting products, oropharyngeal- ment of the object pose estimation, we not only can estimate
swab sampling [3], etc., to further improve the working the correct object pose with RGB-D cameras from mature
efficiency. methods such as DenseFusion [8], [9] and PVN3D [10],
Since robot grasping is the crucial technique for robotic but also directly estimate object pose with RGB cameras by
manipulation, it has long been a significant challenge for employing the method such as OPEPL [11]. Compared with
robotic systems [4], [5]. For various robots, including indus- several similar works, our method utilizes the object pose
trial robots, service robots, and collaborative robots, to accu- to estimate the grasp pose directly. In this way, we propose
rately complete a task, grasping strategy, which dominates the an effective algorithm rather than using the deep learning
reliability of automation, is of significant importance. As a method so that an optimal robotic grasp pose can be quickly
result, related technologies have also been extensively studied determined to perform the reliable grasping task in real time.
for decades, particularly in recent years [4], [7]. However, In this strategy, we can find the 6D grasp pose containing
traditional factory work is mainly concerned with specific the location and angle in 3D space rather than 2D location
grasp locations because the robot arm performs regular work and one orientation by the traditional 2D grasping planner.
that can be pre-defined. Various 6-7 Dof robot arms have been Moreover, our proposed grasping strategy can be utilized in
produced and utilized to accomplish different tasks. With different applications such as collaborative robots, produc-
greater flexibility in the robot arm, grasp estimation also has tion lines, and bin-picking, etc. Also, to achieve the hand-
been widely researched. Several different methods to solve over action in the collaborative mode or production line,
the various tasks have also emerged. The 2D planar grasp is we separate the target object into two parts, i.e., grasping
widely used to solve the task when a target object lies on the area and non-graspable area. Accordingly, the robot arm can
plane. In this scenario, the plane’s height is fixed, so it only find an optimal grasp pose based on the determined grasping
needs to know the position of the object from the camera and area of the target object with any pose for grasping with a
the rotation angle which is vertical to the plane. The problem conventional parallel gripper. More specifically, this paper
here is that the target object is constrained to lie on a plane and makes the following contributions:
the gripper is constrained from one direction. With the recent
1) An algorithm-based optimal grasping strategy is pro-
advance of deep learning-based approaches, a large number
posed to efficiently determine an optimal grasping pose
of methods are used to evaluate the candidates of the oriented
for target objects with any poses based on their 3D
rectangle whether they are reliable to grasp. Although 2D
model during the grasping process using only a RGB
planner grasp provides a more reliable estimate by using a
camera.
neural network, it still has several constraints about the place-
2) The proposed optimal grasping strategy is effective
ment of the gripper and object under different circumstances.
for accomplishing tasks in both collaborative and
With the progression of the 6DoF grasp, a robot arm can
bin-picking modes to grasp objects with various shapes
grasp the target from different angles and locations in the
and sizes.
3D domain. Analytical methods were utilized to analyze the
3) The proposed optimal grasping strategy has a better
geometric structure of the 3D data, and thus the better points
success rate for grasping objects on the YCB-Video
according to different grippers could be found. With the use of
dataset than the state-of-the-art methods.
the 3D cameras with depth sensing such as Microsoft Kinect,
4) Experimental results show that the total computation
Intel Realsense, etc., several methods have been developed to
time required to obtain an optimal grasping pose is
utilize depth information to yield a better grasping capability.
about 0.14 seconds, which reveals the feasibility to pro-
Similarly, the application of deep learning is quite popular
vide real-time operations for practical robot grasping
when searching for a better grasp rectangle from point cloud
applications.
data. However, several problems still exist with this method
since it would be restricted by the angle between objects
and RGB-D camera. Another problem is the precision and II. RELATED WORKS
constraints of the depth camera. Besides estimating the grasp A. 2D GRASPING PLANNER
from 3D information, most of the current grasping methods The grasping strategies in a 2D image plane have been widely
aim at object detection first and then estimating the pose used in circumstances where the target object is placed on
of the object, finally using the predefined grasp pose which a plane. In these circumstances, we only need to consider
is relative to the estimated pose so that the robot arm can the 2D location and one orientation. Recent commonly used
successfully grasp the object. As a result, the problems lie methods can be roughly divided into finding the grasp contact

VOLUME 10, 2022 24057

 H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

point and using the oriented rectangle. Firstly, the method of network PointNet. In 2019, GraspNet [20] used a vibrational
finding the grasp contact point first generates multiple grasp autoencoder and grasp evaluator model to refine the grasps.
points as candidate points and then evaluates the possibility Then REGNet [21] proposed a three-stage approach. First,
of successful grasping through analysis or deep learning- a score network (SN) is used to divide the point cloud
based methods. The study in [12] proposed a method of into regions with high confidence. Then a grasp region net-
estimating from depth maps, but this empirical method gener- work (GRN) will generate several grasp candidates, and use
ally encounters difficulties in dealing with unknown objects. the refinement network to revise the grasp pose to obtain
With the development of deep learning, [13], [14] proposed the final grasp pose. However, to obtain an optimal grasp
a method based on deep learning to solve this problem. First, candidate, these methods based on evaluating the grasp can-
this method generates several grasp candidates from a depth didates on the partial point cloud need to detect the object
map and determines the score of each candidate from the and separate the point cloud of the object from the depth map
network. The one with the highest score will be determined as first.
the final grasp points. Through training on a huge dataset, this
method can handle unknown objects to obtain a grasp point. 2) TRANSFERRING GRASPS FROM EXISTING ONES
Secondly, [15] proposed a method using the oriented rect- This method focuses on transferring the existing grasp to
angle to represent the configuration of the gripper, then select- another, which means finding the correspondences from the
ing a good grasp from the candidates of the oriented rectangle. target to the existing ones if both are in the same cate-
As well as finding candidates from an image directly, [15] gory. [22] proposed a taxonomy-based approach, classifying
estimated the object contour from the depth map to find objects into several categories so that the required grasping
an oriented rectangle with the skeleton of the object. There pattern for a certain object can be found from the associated
are also several methods [17] using deep learning to find categories. [23] also proposed a part-based grasp planning
the oriented rectangle as candidates to determine the best to segment objects into several categories according to their
candidate through another network. shape and volumetric information. The object parts are also
Unfortunately, the methods mentioned above only deal labeled with semantic and grasping information. As a result,
with the situation when the objects are placed on a specific the grasp can be transformed from the same category’s object.
plane. As a result, they suffer from several constraints, includ- DGCM-Net [24] adopted the network to learn a reliable
ing the accuracy of the depth camera, the detection distance of grasp, and then transferred the grasp to unseen objects in the
the depth camera, the situation where the object is occluded, same category. In order to transfer the grasp, both methods
and even the position of the depth camera. need to detect the object and separate the point cloud of the
object from the depth map first. Another method, as proposed
B. 6-DOF GRASPING PLANNER in [25], utilized a model-based registration approach for the
Because of the constraints of the 2D grasping configuration, 6-DoF pose estimation of the target object, and increased
6-Dof grasping has received great attention. The difference the grasping efficiency by reducing the 3D data scanning
from the previously mentioned 2D planner grasp is that the operations.
purpose of this approach is to generate a valid grasp from
the 6D pose space. Through this approach, the robot arm III. OPTIMAL GRASPING STRATEGY
can grasp an object which is placed at any random position In this paper, we propose an optimal grasping strategy utiliz-
and angle and is no longer restricted to objects that must be ing the estimated object pose obtained through an object pose
placed on a fixed plane. The methods of 6Dof grasp can be estimation system for two-finger grippers. Figure 1 illustrates
roughly divided into methods based on partial point cloud the architecture of the proposed grasping process, in which an
and methods based on the complete shape, according to the optimal grasping strategy aims to find an optimal 6Dof grasp
different information used. pose for a two-finger gripper based on an object pose and
preloaded object’s point cloud. With this optimal grasping
1) EVALUATING THE GRASP QUALITIES pose, we can solve the problem of the 2D planner grasp which
OF CANDIDATE GRASPS only can grasp objects placed on a fixed plane. In developing
This method is widely used by sampling many candidate the optimal grasping strategy, the complete point cloud of the
grasps to find the best grasp quality using various algorithms. object is transformed by the object pose so that we would not
Among the traditional methods, [14] proposed a supervised need to consider the limitation caused by the placement angle,
learning approach to find the best grasp pose. With the current accuracy, or sensor noise of the depth camera. At the same
rise of deep learning, several methods have been proposed. time, we expect to find a grasp pose on a specific area of the
GPD [18] used ROI (Region of Interest) to find candidate object so that the collaborative robot would not grasp the area
grasps, and evaluated each candidate by a convolutional neu- currently held by the human hand. As far as production line is
ral network to find the best grasp candidate which had the concerned, we also expect robotic arms can grasp a specific
highest score. Besides, PointnetGPD [19] randomly samples area of the target object for further processing. As a result,
candidate grasps from GPD [18], and evaluates the grasp we propose this optimal grasping strategy to accomplish
quality by direct point cloud analysis with the 3D deep neural these tasks, and Figure 2 shows the flowchart of the optimal

24058 VOLUME 10, 2022

H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

FIGURE 1. Architecture of the proposed grasping process of a robotic manipulation system.

FIGURE 2. Flowchart of the proposed optimal grasping strategy.

grasping strategy. First, the object is transformed by an object

pose. Next, we determine the grasping area for the object by
human assistance as indicated by a dashed rectangle. Third,
the grasping area is segmented into K clusters. Fourth, each
cluster will be processed to create M grasping paths. Finally,
by calculating the cost function of each grasping path, the
system can determine an optimal grasping pose for guiding
the manipulator to complete the grasping task.
FIGURE 3. Complete 3D point cloud of a mug.

A. DETERMINE GRASPING AREA

As mentioned earlier, the proposed strategy aims to find the
optimal grasp solution. Therefore, the robot arm firstly needs
to detect a proper contact area of the object to guarantee a
stable grasping. In order to solve this problem, we roughly
separate each object into grasping and non-graspable areas.
At this stage, we determine a specific area of each object
as the desired grasping area suitable for a specific situation,
task, or application by human assistance with the aid of Mesh-
lab [26], an open source system for processing and editing 3D FIGURE 4. Determined grasping area for the mug in Figure 3.
triangular meshes. Steps to determine the grasping area of a
target object can be summarized as:
1) Use ‘‘Meshlab’’ to open the point cloud file of the used to segment the point cloud of the grasping area into K
object. clusters. K -means is a method of vector quantization, aiming
2) Select the non-graspable area roughly through the to partition n observations into K clusters in which each
‘‘Interactive Selection’’ function in ‘‘Meshlab’’. observation belongs to the cluster with the nearest mean.
3) Delete the selected non-graspable area to keep the With this method, the point cloud of the grasping area can
desired grasping area be easily segmented into K clusters, where K depends on the
4) Save the remaining point cloud as the grasping area. size of the gripper. A bigger gripper has fewer clusters and a
For example, in human-robot collaboration, we might deter- small gripper has more. At the same time, K -means can also
mine a desired grasping area of a target object according find the center of each cluster. As illustrated in Figure 5, the
to the human’s attempt to grasp the object. Take the object determined grasping area is segmented into 3 clusters painted
‘mug’ for example. Figure 3 shows a complete 3D point cloud with different colors, and the corresponding center point is
of the mug, while Figure 4 shows the determined grasping shown by a red point.
area of this object in red dots according to a specific task or
application. C. CREATE GRASPING PATHS
After segmentation, we have K clusters and their centers.
B. SEGMENT INTO CLUSTERS To further find the grasp pose, we create M grasping paths
After determining the grasping area, we can roughly separate in each K cluster. By visualizing every possibility of the
the grasping area from the original point cloud into two grasping paths when we grasp a point or object, we find
areas. However, we want to find more precise areas to fit that there exists a sphere surrounding the grasping target,
the size of the gripper. As a result, K -means method [27] is where the connection from each point on the sphere to the

VOLUME 10, 2022 24059

 H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

obtained by PCA. Besides, the M paths generated through the

Fibonacci sphere contain different directions in the sphere.
Some of the paths will pass through the desktop, depicted
by a blue plane in Figure 8, which is infeasible in reality.
As a result, we can directly delete the paths that will pass
through the desktop by removing them from the candidates.
Figure 8 shows some of the paths that have been removed
from the candidate sets.

FIGURE 5. Segmentation of the grasping area.

grasping point represents a grasping path. Fibonacci Lattice

is a method to create points lying on the surface of the sphere.
As a result, we will use the Fibonacci sphere to create M
grasping paths, where M depends on how many paths we
want to create for grasping paths. The center of the Fibonacci
sphere is the center of each cluster. We then create the M
grasping paths from the M points scattered on the sphere.
In Figure 6, red lines represent all possible grasping paths
FIGURE 7. Main axis of the grasping area.
while setting the Fibonacci sphere on the center of each K
cluster.

FIGURE 6. M paths created by Fibonacci sphere for each cluster. FIGURE 8. Some infeasible candidate paths are removed.

D. CALCULATE GRASPING COST FUNCTION After creating M paths of each cluster as candidates,
To apply this grasping strategy for human-robot interaction an optimal grasping path is required from among all can-
such as collaborative robots or service robots, we expect didates as the best grasp. Thus, the other objective of this
the gripper to grasp the target as a human does. Through work is to estimate an optimal grasping pose for robotic
observation of humans, we find that humans are used to manipulation to successfully grasp the object. The best grasp
grasping an object orthogonally to the main axis of an object. of the target object can be reasonably assumed to contain
For example, most people are used to taking a PET bot- the following features: 1) The grasping path should be as
tle horizontally. Inspired by this human behavior, we use orthogonal to the main axis as possible. 2) There are more
Principal Components Analysis (PCA) to find the main axis points within the radius of the gripper for the gripper to grasp
of the object and grasping area, respectively. PCA is com- and fewer points outside the radius to prevent the gripper
monly used for dimensionality reduction by projecting each from colliding with object. To achieve these objectives, a cost
data point onto only the first few principal components to function is defined to evaluate the cost for each generated
obtain lower-dimensional data while preserving as much as grasping path, which is given by
possible of the data’s variation. Through this dimensionality 0.1 ∗ αp −90 + 0.5 X
N
!
20
reduction method, we can get the main axis by reducing the Jp = ∗ min 1,
2 ,
dimensionality of the object’s point cloud. The red lines in 8 Yip −rk +1
i=1
Figure 7 show the main axis of the grasping area of the object p = 1, . . . , M , (1)

24060 VOLUME 10, 2022

H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

where αp is the angle between path p and the main axis,

Yip represents the distance from points i to grasp path p,
rk represents the radius of the kth cluster, and N represents
the object’s point cloud. The constants in (1) are empirically
obtained. In order to find the grasping path with the two char-
acteristics, Equation (1) contains 2 terms multiplied together,
0.1∗|αp −90|+0.5
where the first term 8 calculates the cost of the
intersection angle αp between path p and the main axis of the
object, and the second term
N
!
X 20
min 1,
2
i=1 Yip − rk + 1
calculates the cost of the distance Yip from all points i to path p
in the object’s point cloud. Through Equation (1), we can find FIGURE 9. Optimal grasp pose produced by the proposed grasping
that the path resulting in a lower value from the cost function strategy.
implies that the path is more orthogonal to the main axis and
it has a higher success rate within the radius of the gripper to
complete the grasping. axis of the grasping area which is produced by PCA to find
The design of Equation (1) is to find the optimal grasp path the 6DoF grasp pose. As illustrated in Figure 9, the coordinate
suitable for the collaborative mode. Furthermore, we might system (a) shows the optimal grasp pose of the object, where
also utilize this strategy for the bin picking mode when the the blue axis represents the best grasping path, which is the
target object is on a flat table. To this end, we need to be con- Z-axis of the gripper, the red axis is the vector of the main
cerned about whether the gripper would collide with the table axis found by PCA which is the X-axis of the gripper, and
or not when the object is lying on the table. Taking this into the last green axis is a vertical vector calculated through the
consideration, Equation (1) can be modified as Equation (2) grasping path and the main axis, which is used as the Y-axis
below: of the gripper. The coordinate system (b) shows the position
N
! and angle of the previous axis of the gripper in the robot arm.
0.1 ∗ αp −90 + 0.5 X 20
Jp = ∗ min 1,
2 On the other hand, we can directly determine the suitable
8 Yip −rk +1 open range of the gripper from rk representing the radius of
i=1
∗T (dk , βp ), p = 1, . . . , M (2) the kth cluster.
T (dk , βp )
 IV. EXPERIMENTAL RESULTS
 (βp −π/2)2
+ 0.05, if dk ≤ g , A. EXPERIMENTAL SETUP
= p = 1, . . . , M ,
 1, 2 otherwise In this paper, we have proposed an optimal grasping strategy
to find an optimal grasp pose for a robotic arm in different
(3)
scenarios. In order to evaluate our proposed optimal grasping
where dk is the distance from the center of the kth cluster strategy, we build a simulation environment to verify the
to the table, βp is the angle between path p and the table, grasping success rate in various tasks.
and g represents the maximum open range of the gripper. The V-REP (simulator CoppeliaSim) [28], an integrated devel-
other values are empirically obtained. To prevent the gripper opment environment, is based on a distributed control archi-
from colliding with the table when grasping the object on tecture: each object/model can be individually controlled via
the table, there is an extra third term expressed in Equation an embedded script, a plugin, an ROS, or BlueZero node, a
(3) to multiply with Equation (1), where dk is used to judge remote API client, or a custom solution. This makes Cop-
whether the grasping point is too close to the table or not. peliaSim very versatile and ideal for multi-robot applications.
Once the grasping point is too close to the table, the term Controllers can be written in C/C++, Python, Java, Lua, Mat-
(βp − π/2)2 /2 + 0.05 in Equation (3) will yield the value lab, or Octave. As a result, we build a simulation environment
of the intersecting angle βj between path p and the table. on V-REP to verify the capability of the proposed optimal
Through Equation (2), we find that a path resulting in a lower grasping strategy.
value from the cost function implies that this path is not only We create a simulation environment shown in Figure 10
more orthogonal to the main axis with a higher success rate (a) that uses only a gripper (RG2), considering only the
within the radius of the gripper to complete the grasping, but end-effector pose (x, y, z, roll, pitch, yaw) to simplify
also prevents the gripper from colliding with the table. the complexity and concentrate on evaluating the proposed
Besides finding the optimal grasping path, we still want approach. We call this simulation environment the collab-
to find the grasp pose for the two-finger gripper. As a result, orative mode, which uses a box to support the target. The
we find a vertical vector of the grasping path and the main box would not be used to calculate the collision with the

VOLUME 10, 2022 24061

 H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

FIGURE 10. Simulation environment with a parallel gripper in (a) the

collaborative mode and (b) the bin picking mode.

FIGURE 12. Determined grasping area (in red points) of the objects in the
gripper after setting it in the simulation environment. Besides, LINEMOD dataset.

we build another simulation environment to simulate grasp-

ing the target object placed on the table with a gripper. We call
this simulation environment the bin picking mode, which is
shown in Figure10 (b).

B. DATASETS
To evaluate the performance of the proposed approach, FIGURE 13. 3D model of objects in the YCB-Video dataset.
we conduct experiments of grasping strategy on two well-
known object datasets, i.e., the LINEMOD dataset [29] and
the YCB-Video dataset [30]. LINEMOD dataset not only
contains the pose of the object but also a 3D model of
each object. There are 13 different objects in the LINEMOD
dataset; ape, bench vise, cam, can, cat, driller, duck, egg
box, glue, hole puncher, iron, lamp, and phone, as shown in
Figure 11. The advantage of adopting the LINEMOD dataset
as our experimental dataset is that each object has a different
shape and size. However, there are several objects which are FIGURE 14. Determined grasping area (in red points) of the objects in the
YCB-Video dataset.
much bigger than the others, such as bench vise, egg box,
hole puncher, and lamp. Thus, this will cause challenges in
grasping. Figure 12 shows the grasping area and original the wood block may be too big for some parallel grippers.
point cloud depicted in red points and blue points, respec- In Figure 14, the determined grasping area and original point
tively. Besides the LINEMOD dataset, we also choose nine cloud for the selected objects are depicted in red points and
objects from the YCB-Video dataset as PointnetGPD [19] blue points, respectively. Note that the grasping areas shown
selected, including cleanser bottle, mug, meat can, tomato in Figs. 12 and 14 are determined for illustration purpose only,
soup can, banana, power drill, mustard bottle, wood block, and can be altered to suit the needs of different applications.
and screwdriver, as shown in Figure 13. Each object also
has its particular shape and size. However, several objects C. EVALUATION METRICS
are much bigger than the others, which would cause more To evaluate the grasping success rate of the proposed optimal
challenges during the grasping experiment. For example, grasping strategy, two different grasping scenarios were con-
ducted in our experiments, called the collaborative mode and
the bin picking mode, respectively. In the collaborative mode,
we aimed to make this experiment as similar as possible to the
real world. First, we let the object randomly rotate and fall
on the box. After moving the gripper to the grasping pose,
which is estimated from the proposed grasping strategy, the
gripper will close its fingers and try to capture the target. Then
the box is removed to simulate that a human is not holding
the object. We call it a grasping success if the object was
still held by the gripper. In the bin picking mode, we built
up a table in the simulation environment, which is shown in
Figure 10 (b). First, we randomly rotate the object and let it
fall on the table. Unlike the collaborative mode experiment,
FIGURE 11. 3D model of objects in the LINEMOD dataset. the gripper would grasp the object and pull it back from the

24062 VOLUME 10, 2022

H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

table to simulate picking objects up from the table in the real

world. If the gripper collides with the table or the object falls
from the gripper, we call it a grasping failure. On the contrary,
if the gripper still held the object after pulling it back, we call
it a grasping success.

D. IMPLEMENTATION DETAILS
Besides designing two different scenarios, we also set up two
different experiments in these two scenarios. First, we ran-
domly rotate the object along its Z-axis and drop it onto the
box or table. Second, we randomly rotate the object along its
X-axis, Y-axis, and Z-axis, then drop the object onto the box
or table. We recorded the success rate through our evaluation
FIGURE 16. Experimental results of the object ‘‘wood block’’ in the bin
method in collaborative mode and bin picking mode. In each picking mode.
of the experiments, we tested ten rounds on each of the objects
in the LINEMOD dataset and YCB-Video dataset.
We have pre-determined each object’s grasping area in shows. Figure 16 shows another experiment process using
two different datasets, which are shown in Figure 11 and the object ‘‘wood block’’ from the YCB-Video dataset in
Figure 13. The cluster K depends on the gripper size and the bin picking mode. Different from the collaborative mode,
size of the grasping area. Besides, we set the parameter M as the gripper performs the actions ‘‘extend’’ and ‘‘pull back’’
256 to create grasp paths of each cluster in these experiments. to simulate picking an object with a robotic arm, as shown
Figure 15 shows the experiment process of the object in Figure 16 (b) and (d), respectively. Besides, if the gripper
‘‘bench vise’’ from the LINEMOD dataset in the collabora- does not collide with the table in the process and still grasps
tive mode. First, an optimal grasp pose is determined from the object after pulling it back from the table, we call it a
the proposed method and the gripper opens according to the success in this experiment which is shown in Figure 16 (d).
diameter of the grasping area found by the method too, which Through our proposed optimal grasping strategy, we not only
is shown in Figure 15 (a). Then the gripper moves to the find the optimal grasp pose but also determine a suitable open
grasp pose and closes the gripper to grasp the object. Finally, range of the gripper according to the diameter of the grasping
the box is removed to simulate that the object is not held by area. Figure 17 shows the experiment on the objects ‘‘mug’’
hand. If the object is still grasped by the gripper, we call and ‘‘woodblock’’ in the YCB-Video dataset. We can see that
it a grasping success in this experiment, as Figure 15 (c) the open range of the gripper in Figure 17 (b) when it grasps
the object ‘‘wood block’’ is much bigger than grasping the
object ‘‘mug’’ shown in Figure 17 (a). Because a suitable
open range of the gripper can be used, collision with the other
parts of the target object can be avoided during the grasping
process. As a result, better performance can be obtained.

FIGURE 17. Open range of the parallel gripper for different objects.

E. COMPARISON RESULTS AGAINST THE

STATE-OF-THE-ART METHODS
For better performance demonstration, the comparison of
the proposed optimal grasping strategy with respect to GPD
and PointnetGPD [19] in grasping is conducted to evaluate
the success rate of different scenarios on the YCB-Video.
FIGURE 15. Experimental results of the object ‘‘bench vise’’ in the As illustrated in Table 1, the proposed optimal grasping strat-
collaborative mode. egy in different scenarios manifests a higher average grasping

VOLUME 10, 2022 24063

 H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

TABLE 1. Comparison between the proposed method and state-of-the-art approaches in the grasping success rate on the YCB-Video dataset.

success rate than those of the state-of-the-art methods. The TABLE 2. Grasping success rate on LINEMOD dataset.
proposed optimal grasping strategy reaches a 95.56% suc-
cess rate in bin picking mode, which is better than Point-
netGPD [19] and GPD [18], and means that our proposed
grasping strategy has a better grasping ability. The proposed
method also reaches a 94.44% success rate in the collabora-
tive mode, which shows the feasible grasping performance for
handing the objects. Figure 17 illustrates the advantage of the
proposed method, where a suitable open range for the gripper
is determined to grasp different objects ‘‘mug’’ and ‘‘wood
block’’ in the YCB-Video dataset. As shown in Figure 17,
we can see that the open range of the gripper is much bigger
when it grasps the object ‘‘wood block’’ in Figure 17 (b) than
the case when it grasps the object ‘‘mug’’ in Figure 17 (a).
Because a suitable open range of the gripper can be used, col-
lision with the other parts of the target object can be avoided
during the grasping process. As a result, better performance
can be obtained as shown in Tables 1 and 2. In addition to
the comparison with the state-of-the-art methods on the YCB-
Video dataset, we also use other objects from the LINEMOD
dataset to evaluate our methods. As illustrated in Table 2, our
proposed optimal grasping strategy also has good grasping
ability in different scenarios. Our methods can reach a 95.38
% success rate and 94.62% success rate in collaborative mode
and bin picking mode, respectively.

F. COMPUTATIONAL EFFICIENCY
OF THE PROPOSED METHOD According to the experimental results, the method to estimate
To show the running speed of the proposed method to obtain the object pose [10] for an object has a running time of
an optimal grasp pose, we need to execute both object pose 0.04 seconds, while the proposed grasping strategy, including
estimation and the proposed grasping strategy. The experi- the steps of ‘Segment into Clusters’, ‘Create Grasping Paths’,
ment of this paper is conducted on a personal computer with and ‘Cost Function’, has a running time of 0.1 seconds on
Intel (R) Core (TM) i7-9700 @ 3.0GHz, an NVIDIA GeForce the above-mentioned platform. Thus, the total computation
RTX 2070 graphic card, and a Logitech C920 webcam. time required to obtain an optimal grasping pose is about

24064 VOLUME 10, 2022

H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

0.14 seconds, which reveals the feasibility to determine an [11] J.-K. You, C.-C.-J. Hsu, W.-Y. Wang, and S.-K. Huang, ‘‘Object pose
optimal grasp pose in real time for practical robot grasping estimation incorporating projection loss and discriminative refinement,’’
IEEE Access, vol. 9, pp. 18597–18606, 2021.
applications. [12] Y. Domae, H. Okuda, Y. Taguchi, K. Sumi, and T. Hirai, ‘‘Fast graspability
evaluation on single depth maps for bin picking with general grippers,’’ in
V. CONCLUSION Proc. IEEE Int. Conf. Robot. Autom. (ICRA), Hong Kong, May/Jun. 2014,
pp. 1997–2004.
This paper has proposed an effective and reliable grasping
[13] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. Aparicio
strategy for determining the optimal grasp pose for widely Ojea, and K. Goldberg, ‘‘Dex-Net 2.0: Deep learning to plan robust
used parallel grippers to deal with target objects with any grasps with synthetic point clouds and analytic grasp metrics,’’ 2017,
arXiv:1703.09312.
pose. To evaluate the performance of the proposed method,
[14] J. Bohg and D. Kragic, ‘‘Learning grasping points with shape
we compare our method with the state-of-the-art GPD [18] context,’’ Robot. Auton. Syst., vol. 58, no. 4, pp. 362–377,
and PointNetGPD [19]. The experimental results show that Apr. 2010.
our proposed method reaches 94.44% and 95.56% success [15] Y. Jiang, S. Moseson, and A. Saxena, ‘‘Efficient grasping from RGBD
images: Learning using a new rectangle representation,’’ in Proc.
rate in collaborative and bin picking mode on the YCB-Video IEEE Int. Conf. Robot. Autom. (ICRA), Shanghai, China, May 2011,
dataset, respectively, outperforming the state-of-the-art meth- pp. 3304–3311.
ods. Besides, experimental results of our evaluation using the [16] M. Vohra, R. Prakash, and L. Behera, ‘‘Real-time grasp pose estimation
for novel objects in densely cluttered environment,’’ in Proc. 28th IEEE
LINEMOD dataset also show that our method is still feasible
Int. Conf. Robot Hum. Interact. Commun. (RO-MAN), New Delhi, India,
in both collaborative and bin-picking modes. Furthermore, Oct. 2019, pp. 1–6.
since the proposed method only requires the object pose [17] Y. Yu, Z. Cao, S. Liang, W. Geng, and J. Yu, ‘‘A novel vision-
estimation from a RGB camera, it can also be applicable based grasping method under occlusion for manipulating robotic
system,’’ IEEE Sensors J., vol. 20, no. 18, pp. 10996–11006,
to other adaptive grippers to develop an optimal grasping Sep. 2020.
strategy. [18] A. T. Pas, M. Gualtieri, K. Saenko, and R. Platt, ‘‘Grasp pose detection
in point clouds,’’ Int. J. Robot. Res., vol. 36, nos. 13–14, pp. 1455–1473,
Dec. 2017.
ACKNOWLEDGMENT [19] H. Liang, X. Ma, S. Li, M. Gorner, S. Tang, B. Fang, F. Sun, and J. Zhang,
The authors would like to thank the National Center for ‘‘PointNetGPD: Detecting grasp configurations from point sets,’’ in Proc.
Int. Conf. Robot. Autom. (ICRA), Montreal, QC, Canada, May 2019,
High-Performance Computing for computer time and facil- pp. 3629–3635.
ities to conduct this research. [20] A. Mousavian, C. Eppner, and D. Fox, ‘‘6-DOF GraspNet: Varia-
tional grasp generation for object manipulation,’’ in Proc. IEEE/CVF
Int. Conf. Comput. Vis. (ICCV), Seoul, South Korea, Oct./Nov. 2019,
REFERENCES pp. 2901–2910.
[1] International Federation of Robotics (IFR). Accessed: Apr. 3, 2021. [21] B. Zhao, H. Zhang, X. Lan, H. Wang, Z. Tian, and N. Zheng, ‘‘REG-
[Online]. Available: https://ifr.org/ Net: Region-based grasp network for end-to-end grasp detection in point
[2] B. Sjøbakk, M. K. Thomassen, and E. Alfnes, ‘‘Implications of automation clouds,’’ 2020, arXiv:2002.12647.
in engineer-to-order production: A case study,’’ Adv. Manuf., vol. 2, no. 2, [22] A. T. Miller, S. Knoop, P. K. Allen, and H. I. Christensen, ‘‘Automatic grasp
pp. 141–149, Jun. 2014. planning using shape primitives,’’ in Proc. IEEE Int. Conf. Robot. Autom.,
[3] Y. Hu, J. Li, Y. Chen, Q. Wang, C. Chi, H. Zhang, Q. Gao, Y. Lan, Z. Li, Taipei, Taiwan, Sep. 2003, pp. 1824–1829.
Z. Mu, Z. Sun, and A. Knoll, ‘‘Design and control of a highly redun- [23] N. Vahrenkamp, L. Westkamp, N. Yamanobe, E. E. Aksoy, and T. Asfour,
dant rigid-flexible coupling robot to assist the COVID-19 oropharyngeal- ‘‘Part-based grasp planning for familiar objects,’’ in Proc. IEEE-RAS 16th
swab sampling,’’ IEEE Robot. Autom. Lett., vol. 7, no. 2, pp. 1856–1863, Int. Conf. Humanoid Robots (Humanoids), Cancun, Mexico, Nov. 2016,
Apr. 2022. pp. 919–925.
[4] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, [24] T. Patten, K. Park, and M. Vincze, ‘‘DGCM-Net: Dense geometrical cor-
E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, ‘‘Scalable deep respondence matching network for incremental experience-based robotic
reinforcement learning for vision-based robotic manipulation,’’ in Proc. grasping,’’ 2020, arXiv:2001.05279.
Mach. Learn. Res., vol. 87, 2018, pp. 651–673. [25] H.-Y. Lin, S.-C. Liang, and Y.-K. Chen, ‘‘Robotic grasping
[5] Y. Hu, X. Wu, P. Geng, and Z. Li, ‘‘Evolution strategies learning with multi-view image acquisition and model-based pose
with variable impedance control for grasping under uncertainty,’’ estimation,’’ IEEE Sensors J., vol. 21, no. 10, pp. 11870–11878,
IEEE Trans. Ind. Electron., vol. 66, no. 10, pp. 7788–7799, May 2021.
Oct. 2019. [26] P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, and
[6] Q. Bai, S. Li, J. Yang, Q. Song, Z. Li, and X. Zhang, ‘‘Object detection G. Ranzuglia, ‘‘MeshLab: An open-source mesh processing tool,’’ in
recognition and robot grasping based on machine learning: A survey,’’ Proc. 6th Eurograph. Italian Chapter Conf., Salerno, Italy, Jul. 2008,
IEEE Access, vol. 8, pp. 181855–181879, 2020. pp. 129–136.
[7] J. Bohg, A. Morales, T. Asfour, and D. Kragic, ‘‘Data-driven grasp [27] X. Jin and J. Han, ‘‘K-means clustering,’’ in Encyclopedia of Machine
synthesis—A survey,’’ IEEE Trans. Robot., vol. 30, no. 2, pp. 289–309, Learning, C. Sammut and G. I. Webb, Eds. Boston, MA, USA: Springer,
Apr. 2014. 2011.
[8] C. Wang, D. Xu, Y. Zhu, R. Martin-Martin, C. Lu, L. Fei-Fei, and [28] The Robotics Simulator CoppeliaSim. Accessed: Apr. 3, 2021. [Online].
S. Savarese, ‘‘DenseFusion: 6D object pose estimation by iterative dense Available: https://www.coppeliarobotics.com/
fusion,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), [29] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige,
Long Beach, CA, USA, Jun. 2019, pp. 3338–3347. and N. Navab, ‘‘Model based training, detection and pose estimation
[9] S.-K. Huang, C.-C. Hsu, W.-Y. Wang, and C.-H. Lin, ‘‘Iterative pose of texture-less 3D objects in heavily cluttered scenes,’’ in Computer
refinement for object pose estimation based on RGBD data,’’ Sensors, Vision—ACCV (Lecture Notes in Computer Science), vol. 7724. Berlin,
vol. 20, no. 15, p. 4114, Jul. 2020. Germany: Springer, 2012.
[10] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun, ‘‘PVN3D: A deep [30] B. Calli, A. Singh, A. Walsman, S. S. Srinivasa, P. Abbeel, and
point-wise 3D keypoints voting network for 6DoF pose estimation,’’ in A. M. Dollár, ‘‘The YCB object and model set: Towards common bench-
Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Seattle, marks for manipulation research,’’ in Proc. Int. Conf. Adv. Robot. (ICAR),
WA, USA, Jun. 2020, pp. 11629–11638. Seattle, WA, USA, Jul. 2015, pp. 510–517.

VOLUME 10, 2022 24065

 H.-H. Chiang et al.: Optimal Grasping Strategy for Robots With Parallel Gripper Based on Feature Sensing

HSIN-HAN CHIANG (Senior Member, IEEE) CHEN-CHIEN JAMES HSU (Senior Member,
received the B.S. and Ph.D. degrees in electri- IEEE) was born in Hsinchu, Taiwan. He received
cal and control engineering from the National the B.S. degree in electronic engineering from the
Chiao-Tung University, Hsinchu, Taiwan, in National Taiwan University of Science and Tech-
2001 and 2007, respectively. He was a Postdoc- nology, Taipei City, Taiwan, in 1987, the M.S.
toral Researcher in electrical engineering from the degree in control engineering from the National
National Taipei University of Technology, Taipei Chiao-Tung University, Hsinchu, in 1989, and the
City, Taiwan, in 2008 and 2009. From August Ph.D. degree from the School of Microelectronic
2009 to January 2017, he was an Assistant Profes- Engineering, Griffith University, Brisbane, QLD,
sor and an Associate Professor with the Depart- Australia, in 1997. He was a Systems Engineer
ment of Electrical Engineering, Fu Jen Catholic University, New Taipei with IBM Corporation, Taipei City, for three years, where he was responsi-
City, Taiwan. From February 2017 to January 2022, he was an Associate ble for information systems planning and application development, before
Professor with the Department of Electrical Engineering, National Taiwan commencing his Ph.D. studies. He joined the Department of Electronic
Normal University, Taipei City. Since February 2022, he has been an Engineering, St. John’s University, New Taipei City, as an Assistant Pro-
Associate Professor with the Department of Vehicle Engineering, National fessor, in 1997, and was appointed as an Associate Professor, in 2004.
Taipei University of Technology. His research interests include the area From 2006 to 2009, he was with the Department of Electrical Engineering,
of intelligent systems and control, autonomous driving systems, vehicle Tamkang University, New Taipei City. He is currently a Professor with the
dynamics and control, and autonomous mobile robots. Department of Electrical Engineering, National Taiwan Normal University,
Taipei City. He is the author or coauthor of more than 200 refereed journals
and conference papers. His current research interests include digital control
systems, evolutionary computation, vision-based measuring systems, sensor
applications, and mobile robot navigation. He is a fellow of IET.

JUN JO received the Ph.D. degree from the Uni-

versity of Sydney, in 1994. He worked as a Post-
doctoral Research Fellow with the Key Centre of
JIUN-KAI YOU received the M.S. degree in Design Computing, University of Sydney, until he
electrical engineering from the National Taiwan joined Griffith University, in 1996. He has been
Normal University, Taipei City, Taiwan, where working on various research projects, including
he is currently pursuing the master’s degree with computer vision and machine learning, and their
the Department of Electrical Engineering. His applications in various areas including robotics,
research interests include computer vision and autonomous cars, drones, the IoTs, satellite data
robotics. analysis, and medical image analysis. He is also
the Program Director of Bachelor of Intelligent Digital Technologies (BIDT)
Degree Program at Griffith University, Australia.

24066 VOLUME 10, 2022