See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/361218581

Design of Pineapple picking manipulator

Thesis · June 2022

DOI: 10.13140/RG.2.2.19273.83049

CITATION READS

1 687

1 author:

Rakibil Hafiz
Hubei University of Technology
2 PUBLICATIONS 1 CITATION

SEE PROFILE

All content following this page was uploaded by Rakibil Hafiz on 10 June 2022.

The user has requested enhancement of the downloaded file.

 来华留学生本科毕业设计（论文）
Undergraduate Graduation Project (Thesis) of an International

Student

题目 Design of Pineapple Picking

Title Manipulator

姓名
Name Rakibil Hafiz （韩非子）
学号
Student ID 1811511116
专业所在学院
School Mechanical Engineering
班级
Class 18.9 Mechanical Engineering
指导教师
Advisor Yan Minxia Laoshi

日期
Time 05 June, 2022
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Abstract

Pineapple harvesting is a physically strenuous and labor-intensive task. It is estimated that the seasonal
agricultural workforce required for pineapple harvesting in the world is more than 1 billion worker hours
annually, accounting for about 15% of the total production cost. The long-term profitability and
sustainability of the apple industry has been eroded due to the decreasing availability and rising cost in
the labor market. Moreover, manual picking activities expose the workers to great risks of ergonomic
injury and musculoskeletal pain, as manual picking involves extensive repetitive body motions and
awkward postures (especially when picking fruits at high locations or deep in the canopy, and
repetitively ascending and descending on ladders with heavy loads). As such, there is an imperative need
for automated pineapple picking to address the aforementioned concerns. In order to tackle this concern,
this paper establishes a 7-DOF robotic device consisting manipulator, end-effector and image-based
vision servo control system was developed that will be used smoothly to harvest pineapples without
hand-labor. The manipulator with 5 DOF PRRRP structure was geometrically optimized to provide
quasi-linear behavior and to simplify the control strategy. The spoon-shaped end-effector with the
pneumatic actuated gripper was designed to satisfy the requirements for harvesting pineapple. The
harvesting robot autonomously performed its harvesting task using a vision-based module. By using a
support vector machine with radial basis function, the fruit recognition algorithm was developed to
detect and locate the pineapple in the trees automatically.

Key words: Pineapple picking manipulator; Harvesting robot; Fruit picking manipulator; 7 DOF
manipulator; End-effector; STC89C52 chip.

I
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

摘 要

菠萝收获是一项体力劳动密集型任务。据估计，全世界菠萝收获所需的季节性农业劳动力
每年超过 10 亿工时，约占总生产成本的 15%。由于劳动力市场的可用性下降和成本上升，苹
果行业的长期盈利能力和可持续性受到侵蚀。此外，手工采摘活动使工人面临着人体工程学伤
害和肌肉骨骼疼痛的巨大风险，因为手工采摘涉及大量重复的身体运动和笨拙的姿势（尤其是
在高处或树冠深处采摘水果时，以及在有重负荷的梯子上重复上升和下降时）。因此，迫切需
要自动采摘菠萝来解决上述问题。为了解决这一问题，本文建立了一个由机械手、末端执行器
和基于图像的视觉伺服控制系统组成的 7 自由度机器人装置，该装置将用于无需人工就能顺利
收获菠萝。对具有 5 自由度 PRRRP 结构的机械手进行几何优化，以提供准线性行为并简化控
制策略。带有气动抓取器的匙形末端执行器是为满足菠萝收获的要求而设计的。收割机器人使
用基于视觉的模块自主执行收割任务。利用径向基函数支持向量机，提出了一种水果识别算法，
用于菠萝的自动检测和定位。

关键词：菠萝采摘机械手；收获机器人；水果采摘机械手；7 自由度机械手；末端执行器；
STC89C52 芯片。

II
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Contents

Abstract ....................................................................................................................... I
摘 要 .......................................................................................................................... II
Chapter 1 INTRODUCTION ........................................................................................... 1
1.1 Background .................................................................................................................. 1
1.2 Aim and Outcome ......................................................................................................... 3
CHAPTER 2 SYSTEM SCHEME DESIGN ........................................................................... 7
2.1 Visual Perception .......................................................................................................... 7
2.2 Manipulator Design ...................................................................................................... 8
2.3 End-effector design ..................................................................................................... 11
Chapter 3 Mechanical System Design Dynamic .......................................................... 14
3.1 Overall structure ......................................................................................................... 14
3.1.1 Structural schematic of automatic seedling picking system ......................................................... 14

3.2 Manipulator................................................................................................................ 15
3.2.1 Analysis of manipulator ................................................................................................................ 15
3.2.2 Compression spring design .......................................................................................................... 18

3.3 Locking mechanism ..................................................................................................... 18

3.4 Seedling guide ............................................................................................................ 19
3.5 Transmission mechanism ............................................................................................ 20
3.5.1 Kinematic analysis of manipulator reciprocating mechanism ...................................................... 21
3.5.2 Simulation analysis ....................................................................................................................... 23
3.5.3 Feeding mechanism...................................................................................................................... 23

3.6 Ratchet ....................................................................................................................... 24

3.7 Harvesting robot system ............................................................................................. 25
3.8 Test materials and equipment ..................................................................................... 26
3.8.1 Parameter measurement of fruit appearance.............................................................................. 26
3.8.2 Stalk tensile test ........................................................................................................................... 26
3.8.3 Compression characteristic test ................................................................................................... 27

3.9 Design of end-effector................................................................................................. 27

3.9.1 Major structure and working principle......................................................................................... 27
3.9.2 Design of fruit stalk clamping mechanism ................................................................................... 28
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

3.9.3 Design of fruit clamping mechanism ............................................................................................ 29

Chapter 4 Mechanical System Design Kinematics ....................................................... 31

4.1 Motion control............................................................................................................ 31
4.1.1 Kinematic model .......................................................................................................................... 31
4.1.2 Controller development ............................................................................................................... 32

4.2 Performance evaluation .............................................................................................. 34

4.2.1 Motion control validation ............................................................................................................. 34
4.2.2 Field test validation ...................................................................................................................... 36

Chapter 5 Design of Control System ........................................................................... 37

5.1 Control system scheme design ........................................................................................................ 37

5.2 Test and analysis ......................................................................................................... 38

5.3 The control system ...................................................................................................... 38
5.3.1 Host computer .......................................................................................................................... 40
5.3.2 Servo Drive Mechanism ............................................................................................................ 40
5.3.3 Manipulator control strategy .................................................................................................... 41
5.3.4 System software design ............................................................................................................ 47

Conclusion ................................................................................................................ 49
Acknowledgement .................................................................................................... 50
Reference.................................................................................................................. 51
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Chapter 1 INTRODUCTION

Picking robot has been gradually applied in the field of agricultural

engineering, it is mainly used for picking tomatoes, cucumbers,
oranges, apples, cherries, pineapples and so on. Picking manipulator
is a 7-DOF manipulator composed of two moving joints and five
rotating joints, it is a multivariable system with highly nonlinear,
strong coupling, and time-varying pose.

1.1 Background

One of the earliest programs in robotic fruit harvesting worldwide

was begun at the University of Florida in the 1980’s by Harrel et al.
(1990b). Subsequently, numerous researchers around the world
studied robotic solutions for fresh market fruit as well as vegetable
harvesting. However, in the scenario of a complicated orchard
environment, the whole process of an intelligent picking operation
is still difficult to tackle. The intelligent fruit picking operation is
mainly divided into three steps: (1) Locating the target fruit by using
the visual system; (2) Automatic path planning combined with target
positioning information; (3) Motion planning combined with the
planned path and completing picking. In addition, the robotic system
needs to be economically approachable as an alternative method to

1
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

hand picking.
Ceres et al. (1998) developed an aided fruit harvesting strategy,
where an operator performed detection of fruits by means of a laser
rangefinder (LRF). The identified position of fruit in spherical
coordinates was used to control the three degrees-of-freedom (DOF)
manipulator for harvesting. d’Esnon (1985); d’Esnon et al. (1987)
developed a vision-based three DOF, hydraulically powered
spherical coordinate manipulator - MAGALI - for golden apple
harvesting, where a monocular camera detected a fruit during a
vertical scan. Subsequently, the telescopic arm translated along the
optical beam until it reached the fruit, which was sensed by a
photoelectric sensor. Ceres et al. (1998); d’Esnon et al. (1987) relied
on open-loop position control, i.e., dead-reckoning, of a robotic
manipulator, thus being vulnerable to fruit motion. The Florida
Citrus Picking Robot by Harrell et al. (1989, 1990a,b), with the goal
of overcoming the limitations of MAGALI, used a closed-loop
camera-in-hand (CiH) configuration along with an ultrasound
transducer for fruit range identification. Computationally
inexpensive contrast-based fruit classification methods used by
Harrell et al. can provide high-frequency image feedback that can
passively compensate for the fruit motion. However, certain outdoor
conditions (e.g., cloudy, or bright day) may require sophisticated
2
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

image processing for efficient classification, thus leaving the system

susceptible to fruit motion. Levi et al. (1988) investigated a vision-
based cylindrical manipulator system for robotic citrus harvesting.
Due to the chosen camera configuration and dead-reckoning during
the reaching stage, the harvesting accuracy is susceptible to
mechanical backlash, bearing wear, and slippage. In Rabatel et al.
(1995); Juste and Sevila (1991), the French and Spanish researchers
proposed a robotic citrus harvesting system called EUREKA. A
Bayesian classifier detected mature fruit from the grayscale images
captured by a monocular vision system. For the spherical
manipulator, the robot motion trajectory was along the straight line
between the camera optical center and the fruit.

1.2 Aim and Outcome

Pineapple harvesting is a physically strenuous and labor-intensive

task. It is estimated that the seasonal agricultural workforce required
for pineapple harvesting in the world is more than 1 billion worker
hours annually, accounting for about 15% of the total production
cost. The long-term profitability and sustainability of the apple
industry has been eroded due to the decreasing availability and rising
cost in the labor market. Moreover, manual picking activities expose
the workers to great risks of ergonomic injury and musculoskeletal

3
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

pain, as manual picking involves extensive repetitive body motions

and awkward postures (especially when picking fruits at high
locations or deep in the canopy, and repetitively ascending and
descending on ladders with heavy loads). As such, there is an
imperative need for automated pineapple picking to address the
aforementioned concerns.
Over the past two decades, several fruit-by-fruit robotic harvesting
systems have been developed. For example, a 7 degree-of-freedom
(DOF) industrial manipulator and a silicone funnel shaped gripper
with an internal camera are developed in. The system scans the
orchard canopy from 40 look-out positions, and for each position the
ripe fruits are detected and picked one-by-one in a looped task. In,
an apple harvesting robot is developed with a global camera, a 7
DOF manipulator, and a finger-based end-effector. Instead of
attaching the camera to the end-effector, the global camera,
independent of other parts of the harvesting system, is employed to
provide a larger field of view. For both robotic systems, a 7 DOF
manipulator is required to approach the fruit. While the 7 DOF
manipulator can provide high maneuverability, it is overly
complicated and extravagant for practical use.

During the pineapple picking process, the manipulator needs to

4
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

approach fruits located at various positions within the work space,

and it thus requires a robust and accurate motion control scheme.
Several advances have been made in manipulator control for robotic
harvesting. For example, a two-step control method is developed in
[12], where the manipulator is initially adjusted such that the
camera’s optical axis points straight to the apple and it is then
controlled to reach out to the apple along the optical axis. This two-
step method can be used in unstructured orchard environments, but
it leads to discontinuous manipulation motion and low harvesting
efficiency. Another control scheme is developed in [15], which
regulates the end-effector along a horizontal path or 45◦ inclined
path to reach the fruits. This scheme is only effective for V-trellis
orchard architectures and would not be suitable for other modern
structured orchards.
Despite the aforementioned efforts, there are still no commercially
available robotic harvesting systems for tree fruits because the
developed systems are still unsatisfactory in performance, too
complicated or expensive to be economically viable, and unreliable
or inefficient for working in the real orchard environment [16,20].
To lay a foundation for automated apple harvesting, this paper
presents the development of a new robotic apple harvesting
prototype. Specifically, a RGB-D camera is exploited as the primary
5
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

sensor, and a deep learning-based perception module is developed

for apple detection and localization. A 3 DOF manipulator and a
vacuum-based end-effector are designed to approach and detach the
apple, respectively. Different from previous studies that rely on high
DOF industrial manipulators, the developed 3 DOF manipulator has
simple and compact structure, making it easier to assemble and
implement apple picking. Furthermore, a motion control scheme is
designed to ensure the manipulator can approach apples with
satisfactory accuracy. Experiments are conducted to illustrate the
performance of the integrated system.
The main contributions of this paper include the following. Firstly,
presenting the synergistic development of a robotic pineapple
harvesting prototype that is simple and compact in structure design.
Secondly, the nonlinear control strategy is designed by fully
exploiting the mechanical structure of the manipulator, which can
accomplish more agile pineapple approaching compared to [12,15].
Finally, the experimental studies validate the design concept and
underpin future research.
The remainder of this paper is organized as follows. Section 2
presents the system design of the apple harvesting robot whereas
Section 3 details the motion control scheme. Experiment results are
provided in Section 4. Finally, conclusions are drawn in Section 5.
6
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

CHAPTER 2 SYSTEM SCHEME DESIGN

The hardware consists primarily of three modules: a color stereo

camera, a 3 DOF manipulator, and a vacuum-based end-effector.
The robot operating system (ROS) is utilized to fully integrate the
entire system and facilitate the communication and control of the
modules. Below is a detailed description of each module.

2.1 Visual Perception

The first and foremost task of automated apple harvesting is the

detection and localization of fruits on the tree. Pineapple detection
is to segment pineapples from the background (i.e., foliage, branches
and trunks), while fruit localization refers to calculating the three-
dimensional spatial position of the detected apples relative to the
camera frame. In this system, A color stereo camera (Bumblebee2,
Point Grey, Vancouver, Canada) with a resolution of 480(H) ×640(L)
at 48FPS was used as the visual system of robot. The stereo vision
system consisted of two identical color CCD sensors was attached
to the top of dual-arm frame to ensure the proper view field, and the
image process was performed by the host computer (Lenovo,
Beijing, China, Inter(R) Core (TM) i3-370 CPU, Random Access
Memory (RAM) 4.0GB). The task of ripe pineapple detection and
localization was completed at two different stages. Firstly, ripe
7
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

pineapple was detected and marked by operator in the 3D

reconstructive image through graphic user interface. Then, the three
dimensions position 𝑃(𝑥, 𝑦, 𝑧) of pineapple object was calculated
through the principle of triangulation (Sun et al., 2011) using the two
images captured by the stereo camera. Since adopting human-robot
collaboration approach to detect and locate ripe pineapple, the
robustness and efficiency of robotic harvesting increased obviously.

Fig. 2.1. The principle of 3d position measurement of pineapple using stereo camera.

2.2 Manipulator Design

Fig. 2.2. CAD model of the proposed 3 DOF manipulator with two revolute joints and
one prismatic joint.

8
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

With the target pineapple locations provided by the perception

system discussed above, a 3 DOF manipulator is then designed and
assembled to efficiently reach the target locations. As illustrated in
Fig. 2.2, the manipulator consists of two revolute joints and one
prismatic joint. The two revolute joints create a pan-and-tilt
mechanism and are affixed to the prismatic base. This design
provides a simple and compact mechanical structure, which not only
offers sufficient DOF for primary pick-and-place tasks but also
facilitates highly efficient motion control.
The pan-and-tilt mechanism is versatile and thus has been widely
used in robotic systems [29,30]. The pan-and-tilt module in this
system contains two revolute joints that are driven by NEMA 23
Teknic ClearPath Servos motors operating at a maximum velocity of
4000 RPM and peak torque of 2 N m. The tilt (vertical) joint is
driven through a 90-degree worm gearbox with an 80:1 ratio and
10.17 N m holding torque, while the pan (horizontal) movement
relies on a parallel shaft gearbox with a 45:1 ratio and 10.02 N m
holding torque. The parallel shaft gearbox is a Molon gearbox
modified to accommodate the motor’s large drive shaft. These two
revolute joints are linked using a 𝐿-shaped aluminum plate, so that
the axes of rotation of the two gearboxes are perpendicular to each
other.
9
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

The velocity of the revolute joints (i.e., servo motors) can be

adjusted via variable frequency pulses ranging from 0 to 500 kHz.
The pulse signals are generated by an Arduino Uno micro-controller.
Based on the serial node provided by the ROS environment,
communications between the Arduino interfaces and the servo
motors are established. Furthermore, to achieve closed-loop control,
the position feedback of the revolute joints needs to be measured.
The position information cannot be accessed through the peripheral
I/O ports of the servo motors, and thus it is necessary to introduce
an adscititious sensing scheme. By default, the motor’s user-settable
counts per revolution gives an exact representation of the distance
that the shaft travels per pulse. Hence, counting the pulses can
deduce the position information of the revolute joints. Based on this
observation, a Teensy 3.6 micro-controller is used as a counter of the
pulse signals, and the real-time position information of the revolute
joints is calculated with the counting results. The Teensy 3.6 micro-
controller is running at the clock rate of 256 MHz, which can provide
an accurate signals counting.
As shown in Fig. 2.2, a prismatic joint is added as the base of the
pan-and-tilt module to extend the depth of the manipulator’s
workspace. Specifically, the prismatic joint is a pneumatically
actuated Lintra rodless air cylinder with a stroke length of 0.61 m
10
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

and a slide carriage. The pneumatic system is driven by a 30-gallon

air compressor, which enables the slide carriage to travel the entire
stroke length in less than one second. High speed is the main
advantage and reason for choosing pneumatic actuation over a screw
based linear stage or a rack-and-pinion system. Moreover, the
Enfield Technologies S2 Valve Positioning System allows for easy
control of the prismatic joint through a standard voltage scheme. The
carriage position along the stroke length can be read from a Balluff
BTL6 MicroPulse transducer, and the control signals are generated
with Arduino and ROS interfaces.
Finally, a hollow aluminum link is installed on the pan-and-tilt
module to make sure that the end-effector can reach the apple
locations. The length and diameter of the link are 0.71 m and 0.04
m, respectively. This link also acts as a vacuum tube for grasping
apple fruits in the harvesting process as referenced in [31].

2.3 End-effector design

Common issues in fruit harvesting end-effector designs include

failure to isolate clustered fruit [32], insufficient gripping strength
[33], low harvesting efficiency from high cycle times [34], and

11
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

damage to the fruit, canopy structures, or the end-effector itself due

to bulky mechanical components [35]. Thus, end-effector design is
a significant challenging task for researchers, and a wide range of
design concepts have been studied with varying degrees of
effectiveness and efficiency. In this system, a vacuum-based end-
effector is utilized. It has been shown that the vacuum-based end-
effector is effective in grasping and detaching tree fruits while
minimizing bruising [36]. Additionally, when the manipulator does
not approach the pineapple accurately, the vacuum-based end-
effector can tolerate the approaching error since it can attract the
fruit within a certain distance when sufficient vacuum flow is
provided. This is important for in-field applications where
unpredictable environmental factors (e.g., instantaneous movement
of fruits due to disturbances from winds and the traveling robot
platform, uneven orchard terrain, etc.) may adversely affect the
system performance (i.e., accuracy in fruit localization and robot
control and movement). Selection of an appropriate diameter of the
end-effector is critical, so that it can achieve a proper flow rate and
vacuum pressure that is needed to grip and detach fruits of different
size, while maintaining the agility of navigating in and out of the tree
canopies. Through preliminary studies, the end-effector diameter of
0.04 m is determined to be adequate for the current robotic system.
12
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Note that most existing vacuum-based end-effectors have much

larger diameter size in order to suck the whole apple into a tubing
system, and thus they cannot reach the apples located deep in the
canopies.
The front end of the end-effector tube is covered with a soft silicone
vacuum cup with a diameter of 0.064 m. Preliminary lab tests
demonstrate that the geometry of vacuum cup and the silicone
material not only ensure that the end-effector can conform to
different contours of pineapples, but also reduce the damage to the
fruit. A rotation mechanism is mounted near the rear end of the end-
effector tube to facilitate the pineapple detaching. Specifically, once
the manipulator reaches the desired pineapple position, the rotation
mechanism is actuated firstly to rotate the whole tube by a certain
angle, and then the manipulator is driven to cut the pineapple.
Furthermore, the rear end of the end-effector tube is connected to a
Craftsman electric powered wet/dry vacuum via a flexible and
expandable tube. During fruit picking, the vacuum machine operates
in continuous mode, which can generate a peak horsepower of 5.5
HP.

13
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Chapter 3 Mechanical System Design Dynamic

Designing the manipulator with the right material is the key feature of this project. A prototype model
of the apple harvesting robot is designed for both efficiency and cost effectiveness. It mainly consists of
an autonomous vehicle, a 5 degree of freedom (DOF) manipulator, an end-effector, the sensors, the
vision system, and control system. Autonomous vehicle is basically a crawler type mobile platform that
will move the whole system.

3.1 Overall structure

The high-speed seedling auto-transplanter is mainly composed of an automatic seedling picking system
and the basket-type planting system (Fig. 1). A ground wheel drives the basket-type planting system,
which completes the transplanting process, and the automatic seedling picking system is responsible for
continuously supplying plug seedlings to the basket-type planting system. The automatic seedling
picking system is symmetric on both sides, which allows for film and double line reciprocating seedling
picking. There is a fixed transmission ratio between the planting system and the automatic seedling
picking system to ensure that the seedlings are accurately fed into the seedling cups. Its specific design
parameters are presented in Table 3-1.

Table 3-1

Main specifications of the machine.

Parameters

Connection method Pull type

Overall size (m × m × m) 3 × 2.37 × 1.6

Overall weight (kg) 1000

Planting rows 2

Plant spacing adjustment range (mm) 250–300

3.1.1 Structural schematic of automatic seedling picking system

The main parts of the automatic seedling picking system include the lifting rod, seedling guide, locking
mechanism, manipulator, frame, pressure plate, adjusting bolt, feeding mechanism, and transmission
mechanism, which have a left–right symmetric arrangement. When the assembly of the manipulator
moves to the discharge position of the plug seedling, the pressure plate presses the seedling rod to the
bottom of the locking mechanism, and the manipulator opens and disperses with the guide groove of the
seedling guide. When the manipulator returns to the seedling picking position, it closes with the guide
groove of the seedling guide until the adjusting bolt and the locking mechanism touch, causing the lifting
rod to lose the restriction of the locking mechanism; at this point, the lifting rod drives the manipulator
14
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

to pick up the seedlings. The seedling feeding mechanism moves step by step, waiting for the next
picking step (Fig 3.1).

3.2 Manipulator

The structure of the manipulator is shown in Fig. 3.2. A compression spring is installed between adjacent
clips, and one end is hinged to the slider. The slider can be moved up and down along the slide bar.
When the slider moves upward, the clip is driven to move upward along the guide rod. At this time, the
clips overcome the elastic force of the compression spring and the pressing force of the limit bearing,
forcing the clips to close and completing the clamping of the plug seedling. When the slider continues
to move upward along the guide rod, the two clips maintain a balanced force under the action of the
compression spring and the limit bearing, and the clamping force remains unchanged until the slider
moves up to the top of the fixed base. When the slider moves down to the bottom of the fixed base, the
limit bearing is exactly tangent to the depression of the clip. At this time, the clip loses the limitation of
the limit bearing and is bounced off by the compression spring. Finally, the plug seedlings are accurately
put into the basket-type planting system.

3.2.1 Analysis of manipulator

The force analysis is conducted when the manipulator clip is closed to pick seedlings (Fig. 3.1). The
spring force of the compression spring, the extrusion force of the limit bearing, and the reaction force of
the stem work together to reach a force balance, and the force analysis is as follows:

Figure 3.1. Seeding

15
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Figure 3.2 Arm

∑𝐹𝑋 = 𝐹𝑆 + 𝐹𝐶 − 𝐹𝑁 = 0
{ ∑𝐹𝑌 = 𝐹𝑃 − 2𝐹𝑓𝑟 − 𝐹𝑓𝑡 − 𝐺 ≥ 0 (3 − 1)
∑𝑀 = 𝐹𝑆. 𝑙𝐴𝐵 − 𝐹𝑁. 𝑙𝐴𝐷 + 𝐹𝐶. 𝑙𝐴𝐸 = 0

where 𝑙𝐴𝐵 is the effective length from point A to B (mm), 𝑙𝐴𝐷 is the effective length from point A to D

(mm), 𝑙𝐴𝐸 is the effective length from point A to E (mm), G is the gravitational force of the plug

seedlings (N), 𝐹𝑓𝑟 is friction between the clamp and seedling stem (N), and Fft is the friction between

the matrix and acupoint disc (N).

16
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

When discharging the seedlings (Fig. 3.3b), the force analysis of the gripper of the manipulator is as
follows:

𝐹𝑆 ′ × 𝑙𝐴𝐵 − 𝐹𝑁 × 𝑙𝐴𝐵 = 0
{ 𝐹𝑓 = 𝜇𝑁 × 𝐹𝑁 (3 − 2)
𝐹𝑁 − 𝐹𝑓 ≥ 0

Figure 3.3. Geometric description of the manipulator. 1. Slider; 2. compression spring; 3. fixed
base; 4. Clips; 5. limit bearing; 6. plug seedling. (a) closing (b) opening.

where FS´ is the reaction force of the compression spring on the clip (N), μN is the coefficient of friction
between the clip and limit bearing, Ff is the force of friction between the clip and limit bearing (N), and
FP is the lifting force of the clips (N).
When the manipulator grips the seedlings, FC, and FC´ are a pair of opposing forces, the force of the plug
seedling is as follows:

2 × 𝐹𝑓𝑟 ≥ 𝐹𝑓𝑡 + 𝐺
{ (3 − 3)
𝐹𝑓𝑡 = 𝜇 × 𝑆
where η is the viscosity coefficient of matrix and plug tray hole (Feng et al., 2013), and S is the contact
area between the matrix and plug tray hole.

The clip dimensions are defined as follows: 𝑙𝐴𝐵 was 22 mm, 𝑙𝐴𝐷 was 98.3 mm, and 𝑙𝐴𝐸 was 120 mm.

17
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

The other measurable parameters were as follows: FC was 23 N, G was 0.3 N, S was 34.2 cm2, and η
was 0.04 N·cm−2. Based on Eq. (1), the pulling force of the manipulator on the plug seedlings FP was
3.34 N, and elastic force FS was 6.5 N.

3.2.2 Compression spring design

When the manipulator reaches the position to pick and discharge the seedlings, the compression spring
acts quickly to achieve the precise clamping and discharging of the plug seedlings. If the spring stiffness
is too small, the clips cannot be opened in time when the manipulator discharges the plug seedlings. As
a result, plug seedlings cannot be fed into the basket-type planting system accurately. If the spring
stiffness is too large, when the manipulator is used to clamp the plug seedlings, a greater lifting force is
required to remove the plug seedlings. Therefore, the design of a reasonable spring structure parameters
is important to improve the precision of the manipulator plug seedlings. Considering that the
compression spring must exhibit a high strength and good performances, a C-grade carbon spring steel
wire was selected. According to the mechanical design manual, the diameter of the compression spring
wire d0 was estimated to be 1 mm, so the bending stress σB was 1800 MPa, and the spring index C was
10. The specific structural parameters of the compression spring can be calculated as follows:

𝜆𝑚𝑎𝑥
𝑡=𝑑+ +∆ (3 − 4)
𝑛
𝐺𝑑 4 𝜆
𝑛=
{ 8𝐹𝐷23 (3 − 5)
[𝜏𝑇] = 0.4𝜎𝐵
𝐷2 = 𝐶. 𝑑
𝐷 = 𝐷2 + 𝑑
0.615 4𝐶 − 1
𝐾= +
𝐶 4𝐶 − 4 (3 − 6)
𝐹𝑆𝐾𝐶
𝑑 = 1.6√
{ [𝜏𝑇]
where the shear modulus G was 80,000 MPa, the maximum deformation λmax was 20 mm, and Δ was
0.29 mm. Eqs. (4)–(6) show that the diameter d of the compression spring wire was 0.5 mm, the mean
diameter D of the spring was 6 mm, the pitch t of the spring was 2.13 mm, and effective working circle
n was 15.

3.3 Locking mechanism

The opening and closing of the manipulator are determined by the locking mechanism, which is
composed of a tension spring, slide panels, a separation rod, a return spring, and closed bearing bases
(Fig. 3.4a). After the manipulator picks up seedlings at point C, the locking mechanism can keep the
manipulator always closed during the C → A process. When the manipulator moves to point A, the
pressure plate presses the lifting rod to the bottom of the slide panels, then the return spring resets the
18
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

separation rod. The manipulator will open all the way during the A → C process, until the adjusting bolt
touches the separation rod at point C. At this time, the seedling lifting rod loses the restriction of the
separation rod, then the lifting rod will be dependent on the tension spring to complete the picking of
the plug seedling (Fig. 3.4b).

Figure. 3.4. Schematic diagram of the locking mechanism. 1. tension spring; 2. slide panels; 3.
separation rod; 4. lifting rod; 5. return spring; 6. closed bearing base; 7. manipulator. (a)
Structure (b) Operation process.

3.4 Seedling guide

When picking seedlings, each plug seedling is required to be within the gripping range of the manipulator.
When seedlings are discharged, each plug seedling can be aligned with the center of the rotating seedling
cup. According to the plug tray parameters, the center distance of the guide groove at the end of picking
and the spacing of guide grooves at the end of discharging can be determined, that is, h = 63.5 mm and
H = 127 mm (Fig. 3.5).
The center lines of the guide grooves aO, bO, cO, and dO extend from point O. The seedling guide uses
OO’ as the axis of symmetry to separate the guide grooves with a certain inclination angle. It is known
from the geometric relationship in Fig. 3.5 that

1 1 1
𝑙𝑂 ′ 𝑑 = 𝑙𝑂 ′ 𝑐 = 𝑙𝑂 ′ 𝑏 = 𝑙𝑂 ′ 𝑎 (3 − 7)
3 5 7

19
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Figure. 3.5. Geometric description of the seedling guide.

Therefore, to realize the equidistant expansion and closing of the seedling picking manipulator, the
inclination angle of the guide groove must satisfy the following:

𝑡𝑎𝑛𝜃1 ∶ 𝑡𝑎𝑛𝜃2 ∶ 𝑡𝑎𝑛𝜃3 ∶ 𝑡𝑎𝑛𝜃4 = 1: 3: 5: 7 (3 − 8)

3.5 Transmission mechanism

The transmission mechanism is mainly composed of the manipulator reciprocating mechanism, the gear
train, the transmission case, and the support frame (Fig. 3.6).
The gear train is installed on the support frame, and the transmission ratio is 1:2. One end of the
20
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

manipulator reciprocating mechanism is connected to the manipulator, and the other end is connected to
the gear train and the output shaft of the transmission case. When the manipulator picks up the plug
seedlings, the transmission case drives the crank to rotate, so that the manipulator picks up and
discharges the plug seedlings back and forth.

Figure. 3.6. Transmission mechanism. 1.manipulator reciprocating mechanism; 2. gear train; 3.

seedling guide; 4. support frame; 5. transmission case.

3.5.1 Kinematic analysis of manipulator reciprocating mechanism

The essential component of the manipulator reciprocating mechanism is the crank slider mechanism.
The crank rotates around point A to C′, the manipulator picks seedlings at point C, and the crank rotates
around point A to point B′. The manipulator then moves to point C′ to discharge the seedlings (Fig. 3.7).
The travel velocity-ratio coefficient K was 1, the extreme position angle was 0°, and there was no sharp
return phenomenon.

21
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Figure. 3.7. Geometric description of the reciprocating mechanism.

According to the movement and installation requirements of the mechanism (Sun et al., 2015), the slider
motion stroke LCC′ was 635 mm. To ensure that the mechanism provided good transmission and had a
minimum structural size, the transmission angle γmin was 40°. The length of the connecting rod can be
obtained from the following formulas:

𝐿𝐶𝐶 = 2𝑙𝐴𝐵 (3 − 9)

(𝑙𝐴𝐵)
𝐿𝐵𝐶 = (3 − 10)
𝑐𝑜𝑠𝛾min
𝐿𝐴𝐵
𝛾= (3 − 11)
𝐿𝐵𝐶
where LAB was 318 mm, LBC was 415 mm, and λ = 0.8 is the connecting rod transmission ratio.
From the geometric relationship in Fig. 3.7, the displacement expression between the slider's movement
position and the farthest extreme position at any time can be obtained by the following formula:

𝐿𝑛 = (𝐿𝐴𝐵 + 𝐿𝐵𝐶 ) − (𝑐𝑜𝑠𝜃1 𝐿𝐴𝐵 + 𝑐𝑜𝑠𝜃2 𝐿𝐵𝐶 ) (3 − 12)

Application of Eqs. (3-11) and (3-12) gives the following equation of motion:
𝛾
𝐿𝑛 = 𝐿𝐴𝐵 [(1 − 𝑐𝑜𝑠𝜃1) + (1 − 𝑐𝑜𝑠2𝜃1)] (3 − 13)
4
The equation of motion of the slider speed v and acceleration a at any time are as follows:

𝑑𝐿𝑛 𝑑𝐿𝑛 𝑑𝜃1 𝜆

𝑣= = . = 𝜔𝐿𝐴𝐵 (𝑠𝑖𝑛𝜃1 + 𝑠𝑖𝑛2𝜃1)
{ 𝑑𝑡 𝑑𝜃1 𝑑𝑡 2 (3 − 14)
𝑑𝑣 𝑑𝑣 𝑑𝜃1
𝑎= = . = 𝜔2 𝐿𝐴𝐵(𝑐𝑜𝑠𝜃1 + 𝜆𝑠𝑖𝑛2𝜃1)
𝑑𝑡 𝑑𝜃1 𝑑𝑡
22
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

3.5.2 Simulation analysis

To determine the output torque of the transmission case, the maximum transplanting frequency was set
to 120 plants·min−1, and the crank speed was 1.57 rad·s−1, which were substituted into Eq. (14) to
simulate the motion state of the slider (Su, 2014). MATLAB was used to calculate the motion state of
the slider at any time (Pan, 2015).
In the interval (0, π), the slider moved away, and when the crank was turned to π, the movement speed
dropped to 0 m·s−1. At this time, the manipulator picked multiple seedlings. In the interval (π, 2π), the
slider reversed movement, and when the crank was turned to 2π, the movement speed dropped to 0
m·s−1 from Fig. 3.8(a). At this time, the manipulator discharged multiple seedlings.
As shown in Fig. 3.8(b), the acceleration of the slider was greater than 0 m·s−2, and the force of the
connecting rod on the slider was greater than the force of friction on the slider. When the crank angle θ1
was , the maximum slider acceleration was 1.25 m·s−2. It is known that the mass of the crank slider
mechanism driving part was 18 kg, and the sliding friction coefficient μ was taken as 0.2. Thus, the
minimum output torque of the transmission case satisfied the following:

𝐹 × 𝑠𝑖𝑛𝜃1 − 𝑓 = 𝑚 × 𝑎 𝑚𝑎𝑥
{ 𝑓 =𝑚×𝑔×𝜇 (3 − 15)
𝑀𝑚𝑖𝑛 = 𝐹 × 𝐿𝐴𝐵
where F is the driving force of the crank (N), f is the frictional force on the slider (N), and Mmin is the
minimum driving force of the transmission case (N·m). According to Eq. (15), because the crank slider
mechanism was symmetrically distributed on the left and right sides, the minimum torque required by
the transmission case was 42.4 N·m.

Figure. 3.8. curve of Crank angle and speed (a) and curve of crank angle and acceleration (b).

3.5.3 Feeding mechanism

The seedling tray conveying mechanism is mainly composed of a pallet, chain cross rod, pressure plate,
ratchet, and underframe (Fig. 3.9). The pallet is fixed on the underframe to support the plug seedling
tray. The power is input by the ratchet, and the drive shaft is driven by the ratchet for intermittent rotation.

23
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

The drive shaft drives the chain cross rod to move, making the plug seedling tray step shift. The platen
above the ratchet is used to keep the seedling tray tightly attached to the pallet to prevent the seedling
tray from shifting when the machine shakes. The seedling tray is placed between the pallet and the platen,
the V-shaped gap of the seedling tray is stuck on the chain cross rods, and the seedling tray can be
stepped on the pallet with the chain cross rods.

Fig. 3.9. Schematic diagram of the feeding mechanism. 1. Pallet; 2. chain cross rod; 3. pressure
plate; 4. Ratchet; 5. underframe.

3.6 Ratchet

The stepping movement of the seedling tray is controlled by a ratchet, and the power is input by the
pawl support, which drives the coaxial sprocket to rotate stepwise, and the chain cross rod drives the
seedling tray to feed longitudinally (Fig. 3.10). The distance that the plug tray moves is equal to the arc
length of the sprocket, and the central angle of the sprocket is equal to the central angle α of the ratchet.
The plug tray has eight holes longitudinally, and the distance between adjacent holes l is 31.75 mm. The
number of ratchet teeth n is 8, and the modulus m is 6. The driving sprocket radius R and sprocket
rotation angle θ satisfies the following conditions:
1
× 2𝜋𝑅 = 𝑙 (3 − 16)
𝑛
360°
𝜃= = 45° (3 − 17)
𝑛
From Eqs. (3-16) and (3-17), the radius R of the sprocket is 40.45 mm. Each time the ratchet moves, the
sprocket rotates 45°, and the plug tray feeds 31.75 mm intermittently.

24
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

3.7 Harvesting robot system

The model of pineapple harvesting robot is shown in Fig.3.10(a,b), which contains a vision unit, a jointed
manipulator, a railed vehicle, a controller, and the end-effector. The vision unit is used to identify and
locate the mature fruits. The controller
receives and processes image data from vision unit, then sends control signal to the manipulator, the
end-effector, and the vehicle through CAN bus. The vehicle carries the whole system moving on the rail,
until receives the stop signal from the controller when the mature fruits were identified in the camera
field. The jointed manipulator positions the end-effector to approach the target fruit, and moves it back
to the container after the fruit is picked by the end-effector.

Figure. 3.10a Pineapple harvesting robot system (3d)

Figure. 3.10a Pineapple harvesting robot system mechanism (2d) {parts: 1. Manipulator 2. End-
effector 3. Fruit container 4. Camera 5. Laser projector 6. Slider unit 7. Human interface 8.
Controller 9. Air pump 10. Vehicle 11. Rail 12. Rail wheel

25
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

3.8 Test materials and equipment

The fruit material used in the test is pineapple. According to the ripeness of pineapple, randomly select
30 fruits in ripe stage, preliminary ripe stage, semi-ripe stage, and ripe stage respectively.

3.8.1 Parameter measurement of fruit appearance

Analyze and summarize the measurement results to obtain the physical parameters, as shown in table 3-
2. The friction coefficient between fruit (or stalk) and different materials is shown in table 3-3.
Table 3-2. Table of physical parameters of pineapple fruit
Maximum Minimum Average

Horizontal diameter(mm)
72.1 58.2 65.5

Vertical diameter(mm)
60.4 48.5 54.2

Table 3-3. Table of friction coefficient

Stainless steel Silica gel Rubber

Fruit 0.48 0.87 0.8

Stem 0.21 0.25 0.43

3.8.2 Stalk tensile test

The test uses 1KN range sensor and the rate of loading is set as 1mm/s. Conducting separation layer
tensile test and calyx tensile test on different ripeness of tomato respectively，get the range of pulling
strength in separation layer is 9.7-28.1N and the average pulling strength is 12.2N; the range of pulling
strength in calyx is 12.5-36.6N and the average pulling strength is 16.2N. Therefore, end-effector uses
separation layer pulling mode to separate fruit and stalk, and the maximum pulling strength is set as 30N.

26
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

3.8.3 Compression characteristic test

The test uses 1KN range sensor and the rate of loading is set as 5mm/s. According to the ripeness, the
fruits are divided to four groups and each group has 30. Then each group is divided into 3 subgroups
randomly and each subgroup has 10. The compression test of different materials will be conducted.
Because the gasbag clamps fruit horizontally during picking process, horizontal compression test is
conducted. From the test, we get the range of damage of pressure force is 60-314N and the average
damage of pressure force is 139N. Therefore, we set the minimum damage of pressure force as 60N.

3.9 Design of end-effector

In robotics, an end effector is the device at the end of a robotic arm, designed to interact with the
environment. The exact nature of this device depends on the application of the robot.
In the strict definition, which originates from serial robotic manipulators, the end effector means the last
link (or end) of the robot. At this endpoint, the tools are attached. In a wider sense, an end effector can
be seen as the part of a robot that interacts with the work environment. This does not refer to the wheels
of a mobile robot or the feet of a humanoid robot, which are not end effectors but rather part of a robot's
mobility.
In my developed pineapple picking manipulator is adjusted to the end part of robotic arm and while
picking fruit the end effector will grab the pineapple and cut it.

3.9.1 Major structure and working principle

The end-effector mainly consists of a fruit stalk clamping mechanism, a fruit clamping mechanism, a
separating mechanism, and a control unit. The major structure is shown in Fig.3.11, where, the base is
fixed on the end of mechanical arm and the relative rotation of fixed tray and rotating tray may achieve
the close and open of clamping pliers. A pressure sensor is pasted between outer layer of gasbag and
fruit, which is used to detect the pressure when clamping fruit. An infrared sensor is equipped on the
bottom of sleeve, which is used to detect whether the fruit is in the sleeve. Hall sensor is installed on the
stroke end of the gear and cylinder, which is used to detect the clamping state of clamping pliers and the
extension state of cylinder.
When the fruits access sleeve, the infrared sensor will detect the fruit and control the air compressor to
inflate the gasbag, and DC motor drives clamping pliers to correctly clamp the stalk. The pressure sensor
may detect the pressure between fruit and gasbag to ensure no damages on fruit on the premise of reliably
clamping fruit. Until getting stability, then drive the double-acting cylinder to take the sleeve to shrink,
in order to cut off and pick the fruit.

27
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

(a) Clamp mechanism opens (b) Clamp mechanism closes

Fig.3.11. Schematic diagram of end-effector.
1. Base 2. Motor fixed plate 3. DC motor 4. Driving gear
5. Driven gear 6. Rotating tray 7. Clamping pliers 8. Pin
9. Toggle clamp 10. Fixed tray 11. Sleeve 12. Gasbag 13. Catch 14. Steel ball bearing 15. Fixed
sleeve 16. Sliding
bearing 17. Clamp spring 18. Double-acting cylinder

3.9.2 Design of fruit stalk clamping mechanism

The schematic diagram of stalk clamping mechanism may be obtained through simplifying each part
according to the working principle, as shown in Fig.3.12 and Fig.3.13. The fixed tray is simplified to
fixed hinge C; rotation of rotating tray is simplified to pole OD and move around fixed hinge O; clamp
pliers is simplified to L-shape pole ACB. The state shown in the Fig.3.12 and Fig.3.13 is stalk clamping
state. Take the torque of C according to static balance relationship:
𝐹𝑐1 𝑙1 − 𝐹𝑑𝑟𝑥 𝑙2 = 0 (3 − 18)
𝐹𝑑𝑟𝑥 = 𝐹𝑑𝑟 𝑠𝑖𝑛𝜃 (3 − 19)
Where, Fcl is the clamping force when clamping. The maximum pulling strength of stalk is 30N and the
friction coefficient between stalk and rubber is 0.43 that is obtained from stem tensile test and friction
measurement. From the formula, we may get the minimum clamping force is 35N. From the geometrical
relationship, we may get θ is 35°. According to the gear rotation relationship:
𝑇𝑑𝑟 = 𝐹𝑑𝑟 𝑙3
𝐹2 𝑑2
𝑇2 = 2
= 𝑇𝑑𝑟 (3 − 20)

𝐹2 𝑑1
𝑇1 = 2

28
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Where, d1 and d2 is the pitch diagram of drive gear and driven gear respectively. After calculating, we
get the driving torque of DC motor should be 1.36N•m, or 13.87kg•cm. So we select ZGA37RG
deceleration DC motor. The pressure is 24V, rated rotating speed is 3.5r/min and rated torque is 15kg•cm.
It may meet the design requirements.

Fig.12. Force diagram of stalk clamping Fig.13. Force diagram of gear transmission
mechanism (left) mechanism(right)

3.9.3 Design of fruit clamping mechanism

Fruit clamping mechanism is shown in Fig.3.14. Three gasbags are distributed in the inner wall of sleeve
evenly between 120°, which may envelop the fruits and keep even force.
The maximum pulling strength of separation layer is 30N upon the stalk tensile test and fruit friction
measurement. The friction coefficient between fruit surface and rubber is 0.87. According to the fruit
compression test, the maximum clamping force is 60N. Supposing the fruit does not contact the bottom
of sleeve in fruit tensile process (no supporting force),

29
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Fig.3.14. Force diagram of fruit clamping

the following expression according to static balance relationship is shown as,

𝐹𝑐1 = 𝐹𝑁𝑦 + 𝐹𝑓 + 𝑚𝑔
𝐹𝑓 = 𝜇𝐹𝑁𝑥
𝐹𝑁𝑥 = 𝐹𝑁 𝑐𝑜𝑠𝜃 (3 − 21)
𝐹𝑁𝑦 = 𝐹𝑁 𝑠𝑖𝑛𝜃
𝐹𝑁 = 𝑃𝑆
The maximum pulling strength of separation layer Fcl takes 30N and the average quality m takes 0.11kg.
The friction coefficient μ between fruit surface and rubber is 0.87. To facilitate calculation, supposing
the gasbag contact area S and the displacement angle θ of each gasbag is equal. It is shown the minimum
clamping force is 8.41N, which is far lower than 60N. Therefore, the minimum clamping force meets
the design requirements. According to the horizontal and vertical diameter, when calculating the contact
area between gasbag and fruit, the fruit may be regarded as a ball, thus the contact area scope is 0.0023-
0.0034m2. The inflation pressure of gasbag P falls between 2.6-3.84KPa. On this basis, we select
CD55B25-45 thin cylinder as the driving part of pulling off the stalk. And the pulling strength reaches
42N under 0.3Mpa pressure.

30
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Chapter 4 Mechanical System Design Kinematics

Kinematic scheme design is the beginning of the mechatronic system design, which is the most
important and difficult for junior students in mechanical engineering education. In this paper, the state
space method with the idea of basic mechanism unit is introduced in kinematic scheme design of Theory
of Machines and Mechanisms Course, which is expressed by state vectors, characteristics matrix and
equations. Based on the basic mechanism unit, a database of the mechanism unit is set up, and computer
aided teaching software of kinematic scheme design is developed for the teaching of Dalian University
of Technology in China.

4.1 Motion control

For a pineapple harvesting system, the manipulator needs to approach the apples located at different
positions with high accuracy and flexibility. To achieve this goal, a motion control strategy is presented
in this section by fully exploiting the mechanical structure of the developed 3 DOF manipulator.

4.1.1 Kinematic model

The kinematical description of the 3 DOF manipulator is shown in Fig. 4.1. Let [𝑥, 𝑦, 𝑧]𝑇 𝜖ℝ3 be the
position of the end-effector expressed in terms of the base frame Fb of the manipulator. According to the
kinematical diagram shown in Fig. 15 and the Denavit–Hartenberg convention [37], the following
forward kinematics function can be obtained:
𝑥 = 𝑑3 cos(𝜃) cos(𝜙) + 𝐷
𝑦 = 𝑑3 sin(𝜙) + 𝑑2 (4 − 1)
𝑧 = −𝑑3 sin(𝜃) cos(𝜙) + 𝑑1
Where, 𝑑1 , 𝑑2 , 𝑑3 𝜖ℝ3 are the link length, and [𝜑, 𝜃, 𝐷]𝑇 ∈ ℝ3 joint variables. The values of link length
and joint variables are listed in Table 4. It is clear that (22) characterizes the position of the end-effector

Fig. 4.1. Kinematical description of the 3 DOF manipulator.

31
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Table 4-1
Model parameters of the 3 DOF manipulator.
Parameter Value
Link 𝑑1 0.0635 m
Link 𝑑2 0.0889 m
Link 𝑑3 0.6985 m
Revolute joint 𝜑 (−25◦,25◦)
Revolute joint 𝜃 (−25◦,25◦)
Prismatic joint 𝐷 (0 m,0.495 m)

as a function of the joint parameters [𝜑, 𝜃, 𝐷]𝑇 . From (4-1) and the facts that 𝜑, 𝜃 ∈ (−25◦, 25◦), it turns
out that
𝑦−𝑑2
𝜙 = arcsin ( 𝑑3
),

𝑑1 −𝑧
𝜃=( ), (4 − 2)
𝑑3 cos(𝜙)

𝐷 = 𝑥 − 𝑑3 cos(𝜃) cos(𝜙),
which characterizes the inverse kinematics to calculate the joint parameters [𝜙, 𝜃, 𝐷]𝑇 from the position
of the end-effector [𝑥, 𝑦, 𝑧]𝑇 . Generally, gradient-based optimization solvers [38,39] can be used to
compute the inverse kinematics of a manipulator. However, since the developed 3 DOF manipulator has
a simple and exploitable structure, its inverse kinematics can be determined by the analytical expression
in (22), which can avoid iterative and complex optimization procedure that is more time consuming and
can induce numerical errors.

4.1.2 Controller development

As described in Section 4.1, the proposed perception algorithm can provide the pineapple locations
expressed in the camera frame. By using conventional calibration techniques [40], the transformation
matrix between the camera frame and the base frame of the manipulator can be determined. Then, the
pineapple location under the manipulator coordinate frame can be calculated via the transformation
matrix. Let [𝑥𝑑 , 𝑦𝑑 , 𝑧𝑑 ]𝑇 ∈ ℝ3 objective is to regulate the end-effector to approach the apple position
[𝑥𝑑 , 𝑦𝑑 , 𝑧𝑑 ]𝑇 from the home position [𝑥0 , 𝑦0 , 𝑧0 ]𝑇 ∈ ℝ3. Note that the revolute joint parameters 𝜑, 𝜃
and prismatic joint parameter 𝐷 are driven by distinct dynamic mechanisms (motor-based and
pneumatic), and hence different control schemes are designed for these two types of joints. Specifically,
the revolute joints 𝜑, 𝜃 are regulated with the velocity-based control method, while the position-based
controller is used to adjust the prismatic joint 𝐷. The velocity-based control method can generate explicit
speed command to smoothly adjust the revolute joints based on real-time position feedback, which is
more accurate and robust than the position-based controller. However, the position feedback of the
current pneumatic prismatic joint is not available, therefore, position-based control method is used for
the prismatic joint.
The velocity-based control scheme for the revolute joints is presented next. Based on (4-1), it can be
found that the end-effector position along the 𝑦𝑏-axis and 𝑧𝑏-axis is determined by 𝜑 and 𝜃. Therefore,
the revolute joints are driven to ensure that [𝑦, 𝑧]𝑇 converges to [𝑦𝑑 , 𝑧𝑑 ]𝑇 . To achieve this task, the
quintic function [37] is exploited to generate a reference trajectory [𝑦𝑟 , 𝑧𝑟 ∈ ℝ2 and then the
32
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

aforementioned regulation problem will be transformed into a trajectory tracking problem.

More precisely, the quintic function-based reference trajectory [𝑦𝑟 , 𝑧𝑟 ]𝑇 has the following form:
𝑦𝑟 (𝑡) = 𝑎𝑦0 + 𝑎𝑦1 𝑡 + 𝑎𝑦2 𝑡 2 + 𝑎𝑦3 𝑡 3 + 𝑎𝑦4 𝑡 4 + 𝑎𝑦5 𝑡 5 ,
𝑧𝑟 (𝑡) = 𝑎𝑧0 + 𝑎𝑧1 𝑡 + 𝑎𝑧2 𝑡 2 + 𝑎𝑧3 𝑡 3 + 𝑎𝑧4 𝑡 4 + 𝑎𝑧5 𝑡^5, (4 − 3)
where 𝑎𝑦𝑖 , 𝑎𝑧𝑖 ∈ ℝ (𝑖 = 0, 1, … , 5) are coefficients of the quintic function and 𝑡 denotes time. Given
[𝑦0 , 𝑧0 ]𝑇 , [𝑦𝑑 , 𝑧𝑑 ]𝑇 , and a time domain [0, 𝑡𝑓 ] , the reference trajectory satisfies
𝑦𝑟 (0) = 𝑦0 , 𝑦𝑟 (𝑡𝑓 ) = 𝑦𝑑 ,
𝑦𝑟̇ (0) = 𝑦𝑟̈ (0) = 𝑦𝑟̇ (𝑡𝑓 ) = 𝑦𝑟̈ (𝑡𝑓 ) = 0,
𝑧𝑟 (0) = 𝑧0 , 𝑧𝑟 (𝑡𝑓 ) = 𝑧𝑑 , (4 − 4)
𝑧𝑟̇ (0) = 𝑧𝑟̈ (0) = 𝑧𝑟̇ (𝑡𝑓 ) = 𝑧𝑟̈ (𝑡𝑓 ) = 0,
Based on (23) and (24), the coefficients [𝑎𝑦𝑖 , 𝑎𝑧𝑖 (𝑖 = 0, 1, … , 5) can be calculated. Note that the
constraints presented in (24) ensure that the initial and final positions of the reference trajectory will be
[𝑦0 , 𝑧0 ]𝑇 and [𝑦𝑑 , 𝑧𝑑 ]𝑇 respectively. Therefore, actuating the revolute joints to make [𝑦, 𝑧]𝑇 follow the
reference trajectory [𝑦𝑟 , 𝑧𝑟 ]𝑇 will lead to the convergence of [𝑦, 𝑧]𝑇 to [𝑦𝑑 , 𝑧𝑑 ]𝑇 . The introduction of
the quintic function-based reference trajectory [𝑦𝑟 , 𝑧𝑟 ]𝑇 also brings several additional advantages. First,
the reference trajectory is continuously differentiable and its terminal velocity and acceleration are zero,
which is conducive to ensuring that the end-effector approaches the desired position along a smooth
path. Second, by adjusting the parameter 𝑡𝑓, the velocity profile of the reference trajectory can be
modified, and thus the end-effector can reach the desired position within a specific time interval. Based
on (4-1), the time derivative of [𝑦, 𝑧]𝑇 can be calculated as:
𝑦̇ = 𝑑3 cos(𝜙) 𝜔𝜙 ,
𝑧̇ = −𝑑3 cos(𝜃) cos(𝜙) 𝜔𝜃 + 𝑑3 sin(𝜃) sin(𝜙) 𝜔𝜙 , (4 − 5)
where 𝜔𝜑, 𝜔𝜃 ∈ ℝ are the angular velocity inputs of the revolute joints 𝜑 and 𝜃, respectively.
𝑇
Furthermore, the error signals [𝑒𝑦 , 𝑒𝑧 ] ∈ ℝ2 constructed as

𝑒𝑦 = 𝑦 − 𝑦𝑟 ,
𝑒𝑧 = 𝑧 − 𝑧𝑟 , (4 − 6)
Based on (25), (26), and by virtue of Lyapunov-based control techniques [41], the velocity controller is
designed as
1
𝜔𝜙 = 𝑑3 cos(𝜙) (−𝑘1𝑒𝑦 + 𝑦𝑟̇ ),

1
𝜔𝜃 = 𝑑 (𝑘2 𝑒𝑧 + 𝑑3 sin(𝜃) sin(𝜙) 𝜔𝜙 − 𝑧𝑟̇ ), (4 − 7)
3 cos(𝜃) cos(𝜙)

where 𝑘1 , 𝑘2 ∈ ℝ+ are positive constant gains. The velocity controller (27) can ensure that the end-
effector position along the 𝑦𝑏-axis and 𝑧𝑏-axis tracks the reference trajectory [𝑦𝑟 , 𝑧𝑟 ]𝑇 asymptotically,
and the detailed stability analysis is given in the following.
Theorem 1. The velocity controller developed in (27) ensures that the end-effector position along the
𝑦𝑏-axis and 𝑧𝑏-axis, i.e., [𝑦, 𝑧]𝑇 , converges to the reference trajectory [𝑦𝑑 , 𝑧𝑑 ]𝑇 asymptotically.

Proof. To prove Theorem 1, a Lyapunov function 𝑉 ∈ ℝ is defined as

1 1
𝑉 ≜ 2 𝑒𝑦2 + 2 𝑒𝑧2 , (4 − 8)

where 𝑒𝑦 and 𝑒𝑧 are the error signals given in (4-6). Based on (4-5) and (4-6), it can be obtained that
33
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

𝑒𝑦̇ = 𝑑3 cos(𝜙) 𝜔𝜙 − 𝑦𝑟̇ ,

𝑒𝑧̇ = −𝑑3 cos(𝜃) cos(𝜙) 𝜔𝜙 + 𝑑3 sin(𝜃) sin(𝜙) 𝜔𝜙 − 𝑧𝑟̇ , (4 − 9)
Taking the time derivative of (4-8) and utilizing (4-7) and (4-9), it can be further derived that
𝑉̇ = 𝑒𝑦 𝑒̇𝑦 + 𝑒𝑧 𝑒̇𝑧
= 𝑒𝑦 (𝑑3 cos(𝜙) 𝜔𝜙 − 𝑦̇𝑟 ) + 𝑒𝑧 (−𝑑3 cos(𝜃) cos(𝜙) 𝜔𝜙 + 𝑑3 sin(𝜃) sin(𝜙) 𝜔𝜙 − 𝑧̇ _𝑟
(4 − 10)
= −𝑘1𝑒𝑦2 − 𝑘2 𝑒𝑧2 ≤0
According to (4-8) and (4-10), the Lyapunov’s stability theorem [41] can be invoked to conclude that 𝑒𝑦
= 0 and 𝑒𝑧 = 0 are asymptotically stable, which indicates that [𝑦, 𝑧]𝑇 converges to [𝑦𝑟 , 𝑧𝑟 ]𝑇
asymptotically.
We next present the position-based scheme for the prismatic joint control. As mentioned in Section 4.2,
the prismatic joint is driven by a pneumatic system. A voltage-based proportional–integral (PI) controller
is utilized to regulate the prismatic joint parameter 𝐷. Specifically, given the desired position
[𝑥𝑑 , 𝑦𝑑 , 𝑧𝑑 ]𝑇 , the inverse kinematics (4-2) is used to calculate the corresponding desired joint
parameters [𝜑𝑑 , 𝜃𝑑 , 𝐷𝑑 ]𝑇 . Then, based on the desired value 𝐷𝑑, the embedded PI controller can adjust
the prismatic joint to ensure 𝐷 converges to 𝐷𝑑.

4.2 Performance evaluation

In this section, comprehensive experiments are reported to demonstrate the performance of the
developed robotic apple harvesting system. We first validate the motion control scheme developed in
Section 4.1 and then evaluate the integrated system in field environments.

4.2.1 Motion control validation

Since different control schemes are employed for the revolute and prismatic joints, their performance is
evaluated separately in the following.

The velocity controller (27) designed for the revolute joints 𝜑 and 𝜃 is tested firstly. We use open-loop
velocity control and position control approaches as the benchmark to facilitate the performance
evaluation. In particular, the open-loop velocity controller is given by

𝑦̇ 𝑟
𝜔𝜙 = 𝑑 ,
3 cos(𝜙)

1
𝜔𝜙 = (𝑑3 sin(𝜃) sin(𝜙) 𝜔𝜙 − 𝑧̇𝑟 ), (4 − 11)
𝑑3 cos(𝜃) cos(𝜙)

Comparing (4-7) with (4-11), it can be found that the developed controller exploits the feedback errors
𝑇
[𝑒𝑦 , 𝑒𝑧 ] to achieve closed-loop control, while the open-loop velocity controller does not include the

feedback error terms. The position control method utilizes the positioning mode provided by the NEMA
23 Teknic ClearPath Servos to rotate the revolute joints. More specifically, given the desired position of
the end-effector, the corresponding desired joint values 𝜑𝑑 and 𝜃𝑑 can be calculated via the inverse
kinematics (4-2), and then the algorithm embedded in the positioning mode is called to regulate the
servo motor towards the desired joint values. To conduct a thorough comparison, four cases are selected.
34
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Under each case, the manipulator is controlled by the aforementioned three methods from the same
home position to a target position. The desired joint values for these four cases are chosen as:

Case 1: [𝜙𝑑 , 𝜃𝑑 , 𝐷𝑑 ]𝑇 = [25°, 15°, 0 𝑚]𝑇 ,

Case 2: [𝜙𝑑 , 𝜃𝑑 , 𝐷𝑑 ]𝑇 = [−6.1°, 22.7°, 0 𝑚]𝑇 ,

Case 3: [𝜙𝑑 , 𝜃𝑑 , 𝐷𝑑 ]𝑇 = [3.7°, −14.3°, 0 𝑚]𝑇 ,

Case 4: [𝜙𝑑 , 𝜃𝑑 , 𝐷𝑑 ]𝑇 = [−12.2°, −1.1°, 0 𝑚]𝑇 .

Note that the prismatic joint parameter 𝐷𝑑 is set as zero and the revolute joint parameters 𝜑𝑑, 𝜃𝑑 are
selected from different quadrants in these four cases, aimed at solely validating the control performance
of the revolute joints. According to (21), the corresponding desired positions for these four cases can be
calculated as follows:

Case 1: [𝑥𝑑 , 𝑦𝑑 , 𝑧𝑑 ]𝑇 = [0.6115 𝑚, 0.31841 𝑚, −0.1003 𝑚]𝑇 ,

Case 2: [𝑥𝑑 , 𝑦𝑑 , 𝑧𝑑 ]𝑇 = [0.6409 𝑚, 0.0144 𝑚, −0.204 𝑚]𝑇 ,

Case 3: [𝑥𝑑 , 𝑦𝑑 , 𝑧𝑑 ]𝑇 = [0.6754 𝑚, 0.1337 𝑚, 0.236 𝑚]𝑇 ,

Case 4: [𝑥𝑑 , 𝑦𝑑 , 𝑧𝑑 ]𝑇 = [0.6825 𝑚, −0.0592 𝑚, 0.0766 𝑚]𝑇 .

Moreover, a spherical marker is attached to the end-effector, and the Qualisys motion capture system is
used to measure the control accuracy. The spherical marker can be detected and localized by the Qualisys
motion capture system stably and precisely, and thus the final position of the end-effector can be
determined via marker identification. Each control method is tested by running 5 times in each case, and
the average distance errors between the final position of the end-effector and the given desired position
are calculated to facilitate the evaluation. The corresponding results are shown in Table 5. It can be seen
that, the proposed velocity control scheme actuates the revolute joints 𝜑 and 𝜃 with higher accuracy for
all four cases compared to the other two methods.

The evaluation of the position control scheme for the prismatic joint 𝐷 is presented next. To separately
test the prismatic joint, the revolute joints 𝜑 and 𝜃 are set as zero while the prismatic joint 𝐷 is actuated
to achieve different desired values. Considering the movement range of the prismatic joint 𝐷 is (0 m,
0.495 m) (see Table 4), these desired values are chosen as

𝐷𝑑 = 0.1 m, 0.2 m, 0.3 m, 0.4 m, 0.495 m.

Table 4-2

Comparison of the distance errors (mm) between the final position and desired position of the
manipulator for different revolute joint control approaches.

Case 1 Case 2 Case 3 Case 4

Open-loop velocity control 8.7 9.1 4.1 4.5

Position control 8.6 4.9 3.8 2.3

Proposed velocity control 2.2 2.6 2.2 1.8

35
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

The movement of the prismatic joint 𝐷 is measured with the aid of spherical marker, and the
measurement results corresponding to the desired values above are 0.0928 m, 0.1932 m, 0.2915 m
0.3904 m, and 0.4855 m. Moreover, the position control scheme can regulate the prismatic joint 𝐷 to
approach the given desired values within one second in different tests, which satisfies the speed
requirement for practical applications.

4.2.2 Field test validation

To evaluate the performance of the integrated robotic pineapple harvesting system, a research of field
tests is shown in this paper.

A total of 97 pineapple picking samples are collected from the humanin-the-loop picking experiments.
In each human-in-the-loop picking experiment, the vision system first identifies and localizes the
pineapples within the manipulator’s workspace. All these pineapples will be shown on the monitor, and
in each harvesting cycle, users need to manually click the screen to choose which apple to pick. Once
the target pineapple is determined, the manipulator is controlled to approach the fruit, and then the
vacuum-based end-effector is actuated to detach the pineapple. Finally, the manipulator returns to the
home position with the detached fruit. Note that in unstructured orchard environments, some pineapples
are surrounded by branches and foliage which obstruct the manipulator’s approaching path. Since the
current perception system cannot detect other objects except for pineapples, users interact with the
system to decide the target fruits, aimed at avoiding obstruction issues. Moreover, the manipulator
follows different inclined angles to reach the pineapple located in diverse regions of the workspace.
Different from [15] which regulates the manipulator along fixed paths to approach a specific area, the
developed control scheme can agilely adjust the manipulator to reach the apple arbitrarily placed in the
workspace, which is a key capability for automated apple picking in structured/unstructured orchard
environments. For the field test, the successful pineapple picking refers to detach the fruit without
causing obvious bruising or surface damage. In all 97 harvesting tests, 80 pineapples are picked
successfully with the picking efficiency being 82.47%. The most frequent cause of missed fruit
detachment is the localization error of the perception system. We find that the pineapple localization
scheme is not robust when the pineapple is partially occluded by leaves. There also exist some cases that
the pineapple is sucked by the end-effector but not successfully detached. This generally occurs when
the pineapple grows on long and flexible shoots. Furthermore, 64 pineapple picking samples are
collected from the fully autonomous picking experiments where the robotic system picks the pineapples
in sequence without any human–robot interaction. Due to obstruction issues mentioned above, 41
pineapples are picked successfully with picking efficiency being 64.06%.

In the field tests, on average, 0.3 s is required to detect and localize all fruits in one image. The durations
for the manipulator to approach an individual pineapple and return back to the home position are
approximately 2.5 s and 2 s, respectively. The fruit detaching usually takes about 4.0 s, which includes
some waiting time that will be reduced in our future systems. The total cycle time required to harvest an
apple is approximately 8.8 s, which is the summation of image processing, manipulation to approach
the pineapple, fruit detaching, and manipulation to return to the home position.

36
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Chapter 5 Design of Control System

The objective is to locate the robot end-effector to the target fruit for harvesting in the presence of
unknown fruit motion. The control objective can be achieved by regulating the time-varying fruit pixel
coordinates to the desired image coordinates, and regulating the end-effector to the desired fruit depth.

5.1 Control system scheme design

As required in fruit picking, the control system is required to detect sensory perceptual system
information and further control execution system. Therefore, the control system uses the mode of
combining single chip and multi-sensor. The single chip selects STC89C52 chip. Pressure sensor selects
and uses FSR402 resistance pressure sensor and its pressure scope is 0.1-10kg. Because pressure sensor
generates analog signal, PCF8951 chip is used for A/D conversion. The correlation-type infrared sensor
selects QT30CM correlation-type PV cell. The working voltage is 5V and the induction distance is 2-
30cm. Hall sensor selects YS44E hall single-pole switch. The structure of control system is as shown in
Fig.5.1. The electrical proportional valve may adjust the voltage to control the pressure of compressed
air into gasbag. The pressure sensor in gasbag may timely detect the pressure and feed the digital signal
to single chip. When reaching the maximum clamping force, single chip control cylinder will pull off
the stalk. It is required to calibrate the pressure monitoring system and get the relationship curve between
pressure and voltage.

Air compressor

Electropneumatic proportional
D/A and power amplifier
valve
Magnetic Valve Cylinder
S
Motor Drive DC motor
C

M Hall sensor

Correlation-type PV sensor

A/D Pressure sensor

Gasbag

Fig.5.1. Structure diagram of control system

37
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

5.2 Test and analysis

The end-effector conducts test for 50 times (50 fruits), among which 43 fruits was successfully held and
separated, 4 fruits unsuccessfully being held and 3 fruits separated from the calyx other than the
separated-layer. So, the average picking success rate is 86%. The principle of picking failure lies in: the
fruit varies largely in shape and the holding force is not enough when clamping smaller fruits; the
separated-layer of unripe fruits is harder broken than calyx. Therefore, accurately sensing the holding
force on fruit and controlling grasping-point on the stem are the mainly future work for improving the
end-effector’s performance.

5.3 The control system

Based on the control objective, a decoupled rotation and nonlinear image-based translation controller is
developed in this section. The fixed camera can view an entire tree canopy and select a fruit to be
harvested. However, the target fruit selected by the fixed camera may not be visible to the CiH; therefore,
the CiH needs to be oriented along the target fruit. Following the orientation, the CiH is regulated to the
desired fruit position using an image-based translation controller. The presented control development
assumes an external nonlinear disturbance, such as wind gust and robot-tree contact, to act on the target
fruit causing unknown fruit motion.
The hardware structure is shown in Fig. 5.2. At the center of the control system was the host computer,
which integrates the control interface and all of software modules to control the whole system. The
sensor signal acquisition system and image acquisition system constituted the input section which was
used to collect external environment information for the harvesting robot. The output section included
a servo driven motor, air pump and end-effector.

38
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

CCD Vision sensor

Signal Modulation Infrared sensor

Circuit Position limited sensor

Collision sensor

Data acquisition module

Drive motor Cutter of the
Host control computer
for cutter end-effector
Electric valve Air pump Gripper of the

end-effector

AC Servo Incremental
Servo drivers
motors and photoelectric
RS232/RS422 1~4
load joints encoder
converters
1~4 1~4

Fig. 5.2. Hardware structure of pineapple harvesting robot

control system.

39
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

5.3.1 Host computer

A Kintek KP-6420i (Kintek Electronics Co., Ltd., Miaoli Hsien, Taiwan, China) industrial computer
with Intel Pentium4 1.7 GHz processor and 512 M memory was selected as the host control computer,
which was responsible for collecting whole sensor signals, processing images online, calculating the
inverse kinematics of manipulator and completing the control algorithm. The host computer transmitted
instructions to the alternating current (AC) servo driver through a serial port to control the joint motors
of waist and arms. HighTek HK-5108 (Shenzhen FangXingLiuTong Industrial Co., Ltd., Shenzhen,
China) RS-232/RS-422 converters were chosen for serial communication functions.

A data acquisition module installed inside host computer was responsible for data

collection from all the sensors except for the vision sensor and output control of the electrical cutter. A
KPCI-847H (Beijing KeRuiXingYe Technology Co., Ltd., Beijing, China) module with 16 channels A/D
and D/A converter was used.

5.3.2 Servo Drive Mechanism

The purpose of motion control was to guarantee that the fruit harvesting robot achieved movement with
an arbitrary angle to grab the object fruit accurately and rapidly. Therefore the key issue was to control
the drive motors of each joint. The closed-loop servo system used to control rotary joints and flexible
joints, consisted of Delta ASDA-AB (Delta Electronics, Inc, Taipei, Taiwan, China) series servo drivers,
ECMA (Delta Electronics, Inc, Taipei, Taiwan, China) series AC servo motors and incremental
photoelectrical encoders, which included the planetary PH (Hubei Planetary Gearboxes Co., Ltd., Wuhai,
China) series reduction gear and the Danaher IDC EC2 (DanaherMontion Co., Ltd., Petaluma, Cal.,
USA) series precise linear electric actuators. To improve the safety of operation, an electromagnetic
brake was fitted to each motor. The ASDAAB series servo driver not only has three control modes
including position control, speed control and torque control, but it also included serial communication
functions for RS485, RS-232 and RS-422. Considering the practical applications of the system, the
rotation joints for the waist, major arm and minor arm employed a position control mode, whilst the
flexible joint of the electric pusher employed a speed control mode. The communication between host
computer and servo drivers of all the joint motors employed the RS-422 mode. Therefore, the frame
computer not only set the parameters and adjusted the gains, but it also monitored the operating state of
the servo driver and alarm conditions. The communication link diagram for the host computer and servo
drivers is shown in Fig. 5.3.

40
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Fig. 5.3. Communication link diagram of the host computer and servo drivers.

5.3.3 Manipulator control strategy

The fruit harvesting robot had an integrated system, comprising environment perception, dynamic
decision making and behavior control. Motion control was the most basic and important ingredient. The
robot vision servo control included two methods, a position-based vision servo (PBVS) and image-based
vision servo (IBVS) (Lippiello, Siciliano, & Villani, 2007; Mariottini, Oriolo & Prattichizzo, 2007).
IBVS was usually used to control the manipulator according to image features and separate vision
reconstruction problems from robot control. This method simplifies robot control and avoids targets
outside the camera visible-field. Thus, it is a commonly usedas a method for robot control. To this end,
in our harvesting robot manipulator control system, the IBVS control method was employed to achieve
location and the picking motion for the target fruit. The structural diagram for fruit harvesting robot
manipulator is shown in Fig. 5.4, where the installed CCD camera is in an eye-in-hand mode.

41
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Fig.5.4. CAD design of the manipulator.

42
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

START

Calculate the differences between centroid

and the center of image: ex and ey

y y y
|𝑒𝑥| > |𝑒𝑦| >
𝑒𝑥max/2 ? 𝑒𝑦max/2 ?
𝑒𝑦
𝑒𝑦 =
8
𝑒𝑦
𝑒𝑦 =
6
y y
|𝑒𝑥| > |𝑒𝑦| >
𝑒𝑥max/16 ? 𝑒𝑦max/12 ? y
𝑒𝑦
𝑒𝑦 =
10 𝑒𝑦
𝑒𝑦 =
10

y y
|𝑒𝑥| > 3? |𝑒𝑦| > 3?
y
𝑒𝑦
𝑒𝑦 = 𝑒𝑦
|𝑒𝑥| 𝑒𝑦 =
|𝑒𝑦|
N N

𝑒𝑦 = 0
𝑒𝑥 = 0

END

Fig. 5.5. Flowchart of small step transformation algorithm.

43
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

At the start of the picking process, the flexible joint contracted in the minor arm during the process of
searching for target fruit. Therefore, the harvesting robot manipulator can be regarded as a three-joint
robot manipulator, and the relationship between camera coordinate system and robot coordinate
system can be obtained according to geometrical relation shown in Fig. 5.6. The camera coordinates
axes (Xc, Yc, Zc) parallel to corresponding axes in robot coordinates (X0, Y0, Z0). L1, L2, L3 are the
lengths of the waist, major arm and minor arm respectively, and 𝜃1, 𝜃2 , 𝜃3 are the joint angles of the
second, third and fourth DOF.

Fig. 5.6. Geometrical relations of manipulator joints.

MATLAB CODE:

l1=1;

l2=1;

l3=2;

%manipulation

% theta d a alpha offset

L1 = Link('d', 0, 'a', 0, 'alpha', pi/2,'offset',0);%m

L1.qlim=[0,pi/2];

L2 = Link('d', -l1, 'a',l2, 'alpha', 0,'offset',0);%m

L2.qlim=[0,pi/2];

L3 = Link('d', 0, 'a', l3, 'alpha', 0,'offset',0);%m

L3.qlim=[0,pi/2];

robot=SerialLink([L1 L2 L3],'name','kinematics'); %m

robot.teach
44
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Pineapples with radius of 400 mm (average radius of the pineapples) were considered as research
objectives. Their projection was a circle on the image captured by video camera. Perspective projection
of a fruit in 3-D space is shown in Fig. 5.7, and formed in the video camera. Feature information of
target fruit in image plane is shown in Fig. 5.8. For a two-dimensional image captured by a video camera,
the origin is a point in the upper right corner. Symbols of u and v denote horizontal and vertical axes
respectively. The image feature of target fruit is characterized as ex and ey, which are the errors between
projection centre coordinate (xg, yg) and image center coordinate (u0, v0). During joint control of
harvesting robot manipulator, image feature of ex varies along with the change of waist joint angle𝜃1,
and image feature of ey varies along with the change of major arm joint angles 𝜃2and minor arm joint
angles 𝜃3.

Fig. 5.7. Perspective projection diagram of fruit in 3-D space.

45
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

O u

(𝑥𝑔 , 𝑦𝑔 )

vv

Fig. 5.8. Location diagram of target fruit in image plane.

It can be seen that the manipulator with 5 DOF PRRRP mechanical structure was geometrically
optimized to simplify the control strategy, and the control algorithm designed to avoid complicated
Jacobian operations. At the same time, the vision systems software gave only planar information of the
target fruit in our robotic system. The distance information between target fruit and camera was unknown.
Hence the manipulator Jacobian could not be directly used in the system.

The process of picking target fruit can be presented as follows. Firstly, each module of harvesting robot
was initialized, and the manipulator made to approach the fruit trees at a proper location. Then the video
obtained image information of target fruit, and the recognition and location were obtained by image
processing software such that the centroid coordinate xg, yg of target in image and the errors ex and ey
obtained by comparison with the image center coordinate u0 and v0.

Secondly, the robot was controlled to move with small step according to the calculated deviations ex
and ey, and eventually it drove them to be zero. Assuming that image plane pixels of video camera are
𝑀 𝑁
M × N, then |𝑒𝑥max| = 2
and |𝑒𝑦max| = 2 . The flowchart of the small step transformation algorithm
is shown in Fig. 12. When the deviations of the small step movements of the waist, major arm and minor
arm were zero, then the centroid of target fruit was coincident with image centre. During the process of
eliminating deviations ex and ey, each joint angle was required to move. This was calculated according
46
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

to Eq. (5-1)

∆𝜃1 = 𝑒𝑥 × ∆𝑑

∆𝜃2 = 𝑘1 × 𝑒𝑦 × ∆𝑑 (5 − 1)

∆𝜃3 = 𝑘2 × 𝑒𝑦 × ∆𝑑

where ∆𝜃1 , ∆𝜃2 , ∆𝜃3 are joint angles of waist, major arm, and minor arm respectively; 𝑘1 , 𝑘2 are the
control parameters of arms; ∆𝑑 is the angle to be adjusted for the movement of a pixel with unit of
degree per pixel.

Then, the host computer sent instructions to the flexible joint to spread. After the object fruit entered
into the gripper of end-effector, the flexible joint stopped spreading. The gripper was then closed and
the electrical cutter cutoff the pineapple stalk.

Finally, the flexible joint backed to its initial position. Thereafter, the gripper was opened and fruit slid
along the flexible tube into the basket.

To achieve continuous picking the above steps were repeated.

5.3.4 System software design

A Windows XP system was employed as an operating platform for its good stability and security. Visual
C++ 6.0 was selected as programming development tool for the host computer. In the system, multiply
tasks needed to be processed simultaneously. Noting that a single-thread might lead to data
communication jams and not guarantee real-time control, a multi-threading event-driven approach was
adopted for the program control system software. The main thread was responsible for the management
of visualization control interface, system initialization; the sub-thread was responsible for
communication and synchronization. The sub-thread system involved video capture, motion control,
elongation test of flexible joint and extraction test of prism sub-threads. The main program flowchart
for the fruit harvesting robot is shown in Fig. 5.9.

47
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Fig. 5.9. Main program flowchart of robot harvesting task.

48
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Conclusion

A self-developed fruit harvesting robot and its control system was developed. The main components of
the robot, i.e., the manipulator, the end-effector, and the image-based vision servo control system, have
been described in detail. The manipulator was geometrically optimized to gain a quasilinear behavior
and simplify the control strategy, and the end-effector with the pneumatic actuated gripper was designed
to satisfy the requirements for the harvesting of pineapples. The harvesting robot autonomously
performed its harvesting task using a vision-based module to detect and localize the apple in the trees,
and control system regulated the manipulator and the end-effector to approach and pick the apples. The
validity of systems was confirmed by performing laboratory tests and field experiments in an open field.

The mechatronic design and motion control of a robotic apple harvesting prototype was presented in
this paper. This prototype integrated a vision-based perception system, 3 DOF manipulator, and vacuum-
based end-effector to execute apple picking. A control scheme was designed to achieve accurate and
agile manipulation motion. Future work will be focused on enhancing fruit detection and localization,
designing optimal path planning scheme for minimizing potential damage to tree canopies by the
manipulator, and extending current system to multiple arms to achieve higher efficiency.

49
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Acknowledgement

This thesis was supported by various research papers on harvesting robots. Also, There was huge support
from my supervisor Yan Minxia Laoshi, who provide me a lot of resources and guidance to finish the
report as well as without whom I couldn’t finish my thesis project. Moreover, I want to give my whole-
hearted thanks to Zhao De-An, Lv Jidong, Ji Wei, Zhang Ying, Chen Yu, Kaixiang Zhang, Kyle Lammers,
Pengyu Chu, Zhaojian Li, Renfu Lu, Changjie Han, Xianwei Hub, Jing Zhanga, Jia You, Honglei Li,
Guohua Wang, Yabo Yu and Qingchun Feng for their research works, since I have learnt a handful of
steps and procedures from their research works.

50
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

Reference

1. Gallardo RK, Galinato SP. 2012 cost estimates of establishing, producing, and packing red
delicious apples in Washington. Washington State University Extension; 2012,
https://pubs.extension.wsu.edu/2012-cost-estimates-of-establishingproducing-and-packing-
red-delicious-apples-in-washington.
2. Fathallah FA. Musculoskeletal disorders in labor-intensive agriculture. Appl Ergon
2010;41(6):738–43, https://doi.org/10.1016/j.apergo.2010.03.003.
3. Reed JN, Tillett RD. Initial experiments in robotic mushroom harvesting.
Mechatronics 1994;4(3):265–79, https://doi.org/10.1016/0957-4158(94)900043.
4. Leu A, Razavi M, Langstädtler L, Ristić-Durrant D, Raffel H, Schenck C, Gräser A, Kuhfuss
B. Robotic green asparagus selective harvesting. IEEE/ASME Trans Mechatron
2017;22(6):2401–10, https://doi.org/10.1109/TMECH.2017. 2735861.
5. Halstead M, McCool C, Denman S, Perez T, Fookes C. Fruit quantity and ripeness estimation
using a robotic vision system. IEEE Robot. Autom. Lett. 2018;3(4):2995–3002,
https://doi.org/10.1109/LRA.2018.2849514.
6. Xiong Y, Ge Y, Grimstad L, From PJ. An autonomous strawberry-harvesting robot: Design,
development, integration, and field evaluation. J Field Robot 2020;37(2):202–24,
https://doi.org/10.1002/rob.21889.
7. Arad B, Balendonck J, Barth R, Ben-Shahar O, Edan Y, Hellström T, Hemming J, Kurtser P,
Ringdahl O, Tielen T, van Tuijl B. Development of a sweet pepper harvesting robot. J Field
Robot 2020;37(6):1027–39, https://doi.org/10.1002/ rob.21937.
8. Peterson DL, Bennedsen BS, Anger WC, Wolford SD. A systems approach to robotic bulk
harvesting of apples. Trans ASAE 1999;42(4):871–6, https://doi. org/10.13031/2013.13266.
9. De Kleine ME, Karkee M. A semi-automated harvesting prototype for shaking fruit tree limbs.
Trans ASABE 2015;58(6):1461–70, https://doi.org/10.13031/ trans.58.11011.
10. He L, Fu H, Sun D, Karkee M, Zhang Q. Shake-and-catch harvesting for fresh market apples
in trellis-trained trees. Trans ASABE 2017;60(2):353–60, https: //doi.org/10.13031/trans.12067.
11. He L, Zhang X, Ye Y, Karkee M, Zhang Q. Effect of shaking location and duration on
mechanical harvesting of fresh market apples. Appl Eng Agric 2019;35(2):175–83,
https://doi.org/10.13031/aea.12974.
12. Baeten J, Donné K, Boedrij S, Beckers W, Claesen E. Autonomous fruit picking machine: A
robotic apple harvester. In: Laugier C, Siegwart R, editors. Field and service robotics. Springer;
2008, p. 531–9.
13. De-An Z, Jidong L, Wei J, Ying Z, Yu C. Design and control of an apple harvesting robot.
Biosyst Eng 2011;110(2):112–22, https://doi.org/10.1016/j. biosystemseng.2011.07.005.
14. Davidson JR, Silwal A, Hohimer CJ, Karkee M, Mo C, Zhang Q. Proof-of-concept of a robotic
apple harvester. In: Proc. IEEE/RSJ int. conf. intell. robots syst.. Daejeon, Korea; 2016, p. 634–
9, https://doi.org/10.1109/IROS.2016.7759119.
15. Silwal A, Davidson JR, Karkee M, Mo C, Zhang Q, Lewis K. Design, integration, and field
evaluation of a robotic apple harvester. J Field Robot 2017;34(6):1140–59,
https://doi.org/10.1002/rob.21715.
16. Gongal A, Amatya S, Karkee M, Zhang Q, Lewis K. Sensors and systems for fruit detection
and localization: A review. Comput Electron Agric 2015;116:8–19,
https://doi.org/10.1016/j.compag.2015.05.021.
51
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

17. Plebe A, Grasso G. Localization of spherical fruits for robotic harvesting. Mach Vision Appl
2001;13(2):70–9, https://doi.org/10.1007/PL00013271.
18. Sa I, Ge Z, Dayoub F, Upcroft B, Perez T, McCool C. Deepfruits: A fruit detection system using
deep neural networks. Sensors 2016;16(8):12–22, https: //doi.org/10.3390/s16081222.
19. Lu R, Pothula AK, Mizushima A, VanDyke M, Zhang Z. System for sorting fruit. 2018,
9919345.
20. Lu R, Zhang Z, Pothula AK. Innovative technology for apple harvest and in-field sorting. Fruit
Qtly 2017;25(2):11–4.
21. Intel RealSense™ camera D400 series product family datasheet. 2020,
https://www.intelrealsense.com/wp-content/uploads/2020/06/Intel-RealSenseD400-Series-
Datasheet-June-2020.pdf.
22. He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: Proc. IEEE int. conf. comput. vision.
Venice, Italy; 2017, p. 2961–9, https://doi.org/10.1109/ICCV. 2017.322.
23. Barea R, Bergasa LM, Romera E, López-Guillén E, Perez O, Tradacete M, López J. Integrating
state-of-the-art CNNs for multi-sensor 3D vehicle detection in real autonomous driving
environments. In: Proc. IEEE intell. transp. syst. conf.. Auckland, New Zealand; 2019, p. 1425–
31, https://doi.org/10.1109/ITSC.2019. 8916973.
24. Vuola AO, Akram SU, Kannala J. Mask-RCNN and U-net ensembled for nuclei segmentation.
In: Proc. IEEE int. symp. biomed. imaging. Venice, Italy; 2019, p. 208–12,
https://doi.org/10.1109/ISBI.2019.8759574.
25. Ganesh P, Volle K, Burks TF, Mehta SS. Deep orange: Mask R-CNN based orange detection
and segmentation. IFAC-PapersOnLine 2019;52(30):70–5, https:
//doi.org/10.1016/j.ifacol.2019.12.499.
26. Dutta A, Zisserman A. The VIA annotation software for images, audio and video. In: Proc. ACM
int. conf. multimed.. New York, NY, USA; 2019, p. 2276–9,
https://doi.org/10.1145/3343031.3350535.
27. Chu P, Li Z, Lammers K, Lu R, Liu X. DeepApple: Deep learning-based apple detection using
a suppression mask R-CNN. Pattern Recognit Lett 2021;147:206–11,
https://doi.org/10.1016/j.patrec.2021.04.022.
28. Hartley R, Zisserman A. Multiple view geometry in computer vision. Cambridge University
Press; 2003.
29. Murray RM, Li Z, Sastry SS. A mathematical introduction to robotic manipulation. CRC press;
1994.
30. Lee Y, Lan C, Chu C, Lai C, Chen Y. A pan–tilt orienting mechanism with parallel axes of
flexural actuation. IEEE/ASME Trans Mechatron 2013;18(3):1100–12,
https://doi.org/10.1109/TMECH.2012.2195192.
31. Lu R, Li Z, Lammers K. System and method for harvesting fruit. 2020, 62982833.
32. Kondo N, Yata K, Iida M, Shiigi T, Monta M, Kurita M, Omori H. Development of an end-
effector for a tomato cluster harvesting robot. Eng Agric Env Food 2010;3(1):20–4,
https://doi.org/10.1016/S1881-8366(10)80007-2.
33. Bamotra A, Walia P, Prituja AV, Ren H. Fabrication and characterization of novel soft compliant
robotic end-effectors with negative pressure and mechanical advantages. In: Proc. int. conf. adv.
robot. mechatronics. Singapore; 2018, p. 369–74,
https://doi.org/10.1109/ICARM.2018.8610688.
34. Davidson JR, Hohimer CJ, Mo C. Preliminary design of a robotic system for catching and

52
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

storing fresh market apples. IFAC-PapersOnLine 2016;49(16):149–54,

https://doi.org/10.1016/j.ifacol.2016.10.028.
35. Hemming J, Tuijl B, Gauchel W, Wais E. Field test of different end-effectors for robotic
harvesting of sweet-pepper. Acta Hortic 2016;1130:567–74,
https://doi.org/10.17660/ActaHortic.2016.1130.85.
36. Gerber CE. Fruit harvesting machine. 1985, 4501113.
37. Siciliano B, Sciavicco L, Villani L, Oriolo G. Robotics: modelling, planning and control.
London: Springer-Verlag; 2010.
38. Wang L-C, Chen C-C. A combined optimization method for solving the inverse kinematics
problems of mechanical manipulators. IEEE Trans Robot Autom 1991;7(4):489–99,
https://doi.org/10.1109/70.86079.
39. Sugihara T. Solvability-unconcerned inverse kinematics by the Levenberg– Marquardt method.
IEEE Trans Robot 2011;27(5):984–91, https://doi.org/10. 1109/TRO.2011.2148230.
40. Bouguet J-Y. Camera calibration toolbox for MATLAB. 2013, http://www.vision.
caltech.edu/bouguetj/calib_doc/.
41. Khalil HK. Nonlinear systems. Upper Saddle River, NJ: Prentice-Hall; 2002.
42. Chen Jun, Wang Hu, and Other. (2012). Transactions of the Chinese Society of Agricultural
Machinery. Transactions of the Chinese Society of Agricultural Machinery, 43(10), 151-154.
43. Jin Bo, Lin Longxian. (2014). Design and Force Control of an Underactuated Robotic Hand for
Fruit and Vegetable Picking. Journal of Mechanical Engineering, 50(19),1-8.
44. Ling P, Ehsani R, and Other. (2004).Sensing and end-effector for a robotic tomato harvester.
2004 ASABE Annual Meeting, ASABE Paper 043088.
45. Li Guoli, Ji Changying, and Other. (2014). Research progress and analysis of end-effector for
fruits and vegetables picking robot. Journal of Chinese Agricultural Mechanization, 35(5), 231-
236.
46. Li Zhiguo, Liu Jizhan, Li Pingping. (2010). Relationship between mechanical property and
damage of tomato during robot harvesting. Transactions of the CSAE, 26(5), 112-116.
47. Liu Jizhan, Li Pingping, and Other. (2008). Hardware Design of the End-effector for Tomato-
harvesting Robot. Transactions of the Chinese Society of Agricultural Machinery, 39(3), 109-
112.
48. Monta M, Kondo N, Ting K C. (1998). End-effectors for tomato harvesting robot. Artificial
Intelligence Review,12(1-3),11-25.
49. Naoshi K, Koki Y, Michihisa I, and Other.
50. (2010).Development of an end-effector for a tomato cluster harvesting robot. Engineering in
Agriculture, Environment and Food, 3(1), 20-24.
51. Ma Lvzhong, Yang Wenliang, and Other. (2009). Structure Design and Experiment of the End-
effector for Appleharvesting Robot. Journal of Agricultural Mechanization Research,(12),65-
67.
52. Tu Hongling, Ji Changying. (2010). Design of an endeffector for a multi-use picking robot.
Acta Agriculturae Zhejiangensis, 22(3),370-373.
53. Zhang Fanian, Li Zhen, and Other. (2014). Study on Kiwifruit Physical Parameters and Damage
Factors. Journal of Agricultural Mechanization Research, (11), 141-145.
54. Zhang Kailiang, Yang Li, and Other. (2012). Design and Experiment of elevated Substrate
Culture Strawberry Picking Robot. Transactions of the Chinese Society of Agricultural
Machinery, 43(9),165-171.

53
 毕业设计（论文）专用纸
Specified paper for graduation project (thesis)

55. Bulanon, D. M., Kataoka, T., & Ota, Y. (2002). A segmentation algorithm for the automatic
recognition of Fuji apples at harvest. Biosystems Engineering, 83(4), 405-412.
56. Bulanon, D. M., Kataoka, T., & Okamoto, H. (2004). Development of a real-time machine
vision system for the apple harvesting robot. Sapporo, Japan Annual conference of society of
instrument and control engineers (SICE) (pp. 595-598).
57. Edan, Y., Rogozin, D., Flash, T., & Miles, G. E. (2000). Robotic melon harvesting. IEEE
Transactions on Robotics and Automation, 16(6), 831-835.
58. Foglia, M. M., & Reina, G. (2006). Agricultural robot for radicchio harvesting. Journal of Field
Robotics, 23(6), 363-377.
59. Grand, d’E., Rabatel, A. G., Pellenc, R., Journeau, A., & Aldon, M. J. (1987). Magali: A self-
propelled robot to pick apples. American Society of Agricultural Engineering Paper, 87-107.
60. Hancock, J. (1999). Cultural systems, strawberries. NY, USA: CABI Publishing.

54

View publication stats