SPECIAL SECTION ON DATA MINING FOR INTERNET OF THINGS

Received August 12, 2020, accepted August 25, 2020, date of publication September 18, 2020, date of current version October 13, 2020.
Digital Object Identifier 10.1109/ACCESS.2020.3024891

An AIoT Based Smart Agricultural System for

Pests Detection
CHING-JU CHEN1 , YA-YU HUANG2 , YUAN-SHUO LI2 , CHUAN-YU CHANG 3, (Senior Member, IEEE),
AND YUEH-MIN HUANG 2 , (Senior Member, IEEE)
1 Department of Bachelor Program in Interdisciplinary Studies, National Yunlin University of Science and Technology, Douliou 64002, Taiwan
2 Department of Engineering Science, National Cheng Kung University, Tainan 70101, Taiwan
3 Department of Computer Science and Information Engineering, National Yunlin University of Science and Technology, Douliou 64002, Taiwan
Corresponding author: Yueh-Min Huang ([email protected])
This work was supported in part by the Ministry of Science and Technology, Taiwan, R.O.C., under Grant MOST 108-2321-B-067F-001-,
and in part by the Intelligent Recognition Industry Service Center through the Featured Areas Research Center Program within the
framework of the Higher Education Sprout Project by the Ministry of Education (MOE), Taiwan.

ABSTRACT In this study, artificial intelligence and image recognition technologies are combined with
environmental sensors and the Internet of Things (IoT) for pest identification. Real-time agricultural
meteorology and pest identification systems on mobile applications are evaluated based on intelligent pest
identification and environmental IoT data. We combined the current mature AIoT technology and deep
learning and applied it to smart agriculture. We used deep learning YOLOv3 for image recognition to obtain
the location of Tessaratoma papillosa and analyze the environmental information from weather stations
through Long Short-Term Memory (LSTM) to predict the occurrence of pests. The experimental results
showed that the pest identification accuracy reached 90%. Precise positioning can effectively reduce the
amount of pesticides used and reduce pesticide damage to the soil. The current research provides the location
of the pest and the extent of the pests to farmers can accurately use pesticide application at a precise time and
place and thus reduce the agricultural workforce required for timely pest control, thus achieving the goal of
smart agriculture. The proposed system notifies farmers of the presence of different pests before they start
multiplying in large numbers. It improves overall agricultural economic value by providing appropriate pest
control methods that decrease crop losses and reduce the environmental damage caused by the excessive
usage of pesticides.

INDEX TERMS Deep learning, YOLOv3, pests and diseases, smart agriculture, unmanned aerial vehicle
(UAV), artificial intelligence (AI), Internet of Things (IoT), the artificial Intelligence of Things (AIoT).

I. INTRODUCTION Crops often suffer from bacterial infections due to pests

Crop production is closely related to the presence of pests and that result in large-scale crop diseases. To prevent such con-
plant diseases. Pests often hide behind the leaves of plants ditions, it is necessary to burn the infected crops to prevent
during the day to avoid the heat and appear on the leaves in the spread of the bacteria. However, this approach causes
the evening or at night. Therefore, it is not easy to observe significant damages in agricultural production without effec-
their presence on crops during the day. When farmers become tively resolving pest problems. Therefore, the goal of this
aware of pest damage, pests have often multiplied and spread research is to apply the AIoT and deep learning technology to
uncontrollably. At this stage, a large quantity of pesticides an environmental analysis of crop growth and the prediction
is required to spray the crops to eliminate the pests and of pest occurrence to improve the productivity of crops and
reduce agricultural damage. However, once the crops have reduce the size of the agricultural workforce.
been sprayed with pesticides in the growing season, pesticide The application of image recognition has become more
residues remain even after washing. extensive due to the development of artificial intelligence in
image processing models. Gladence et al. [1] applied AIoT
to home automation recommendations. In another work by
The associate editor coordinating the review of this manuscript and Priya et al. [2], the common image recognition CNN model
approving it for publication was Patrick Hung. was used in medical detection and prediction.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
180750 VOLUME 8, 2020
C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

In recent years, image recognition technology has been In summary, supervised learning adds artificial labels to
used to assist with pest recognition for the purpose of control- the input data and uses regression analysis methods to obtain
ling agricultural pest damage and increasing crop production. the predicted results. Unsupervised learning finds suitable
Mary Gladence et al. [3] proposed the use of human-robot patterns from a large amount of data through algorithms and
interaction in the field of artificial intelligence and provided classifies the data automatically.
a reference for pest identification through a combination of
environmental factor information and deep learning to help B. DEEP LEARNING TO IDENTIFY AND CLASSIFY PEST
farmers analyze crop growth trends and prevent pest damage IMAGES
early Deep learning is based on the machine learning framework,
so the training process first involves unsupervised learning
II. RELATED WORKS and clusters the training data set to learn what sort of data
Martin and Moisan [4] established an automated system for will be classified. Supervised learning is then performed to
pest identification. In their system, the camera is placed above label the expected output value of each entry with the feature
a sticky insect trap and automatically captures images of vectors in the training data set given as input and the expected
pests collected by the insect trap each day. The Single Seed classification given as the output. Finally, the loss function is
Descent (SSD) method was used to identify and analyze used to calculate the standard deviation between the expected
the pests in the images. The collected data has been used output and the actual output.
to develop appropriate pest control methods based on the There are two common approaches in deep learning for
identified pest species. Once the SSD model has been trained, pest identification and classification:
the pests identification accuracy reaches 84%, and the pest
1) Miranda et al. [6] used the VGG19 method for image
species classification accuracy reaches 86%.
feature extraction and recognition in the detection
Wang et al. [5] set up a large number of environmental
of 24 types of pests from crop images.
sensors in an apple orchard for the purpose of recording
2) The authors in [7] detected and classified 12 different
the status of the orchard. The YOLOv3 model was used to
pests using several CNN approaches and compared the
identify anthrax on the surface of the apples and analyze the
classification results with machine learning methods
health of the apples. The authors also used the YOLOv3-
such as SVM and Fisher. The classification accuracy
Dense model, which is more suitable for identifying apple
of the machine learning method was 80%, and the
anthracnose.
classification accuracy rate of the CNN method was
In the above work, the YOLOv3 and YOLOv3-Dense mod-
95%.
els were both used to identify apple anthracnose, and the SSD
was successfully used to identify pests.
C. USING IMAGE AUGMENTATION TO INCREASE THE
PESTS TRAINING SAMPLE DATABASE
A. MACHINE LEARNING TO IDENTIFY AND CLASSIFY
PEST IMAGES
Data augmentation methods can be roughly divided into
Geometric Transformation and Photometric Transformation
With the advancement of science and technology, the amount
methods. Ding and Taylor [8] studied the impact of different
of image data is increasing, along with the time required to
data augmentation methods on image recognition rate. Three
classify the image data. Therefore, scholars are studying how
geometric transformations (flipping, rotating, and cropping)
to use machine learning to recognize and classify images.
and three luminosity transformations (color jittering, edge
Machine learning is divided into supervised learning and
enhancement, and fancy principal component analysis). The
unsupervised learning, the main difference being whether or
amplified image samples and the original image samples are
not the machine is able to automatically extract features from
used as training data for training on the CNN model. The
the data structure.
experimental results showed that each amplification method
improves the accuracy of CNN, where the impact of cropping
1) SUPERVISED LEARNING is the most obvious. Therefore, the authors speculate that the
During the machine training process, it is necessary to training samples generated by the cropping method will avoid
provide machines with labeled data. For example, after a the over-fitting of data and improve CNN performance.
machine has seen 1,000 labeled images of apples and oranges, In addition to augmenting the images by geometric and
we can give a test image and asked whether the image con- photometric transformations, it is also possible to transform
tains apples or oranges. the image style through the CycleGAN [9] or transfer the
features of objects to other objects to increase the number
2) UNSUPERVISED LEARNING of training samples. The aforementioned apple anthracnose
There is no need to label the data in advance, and the machine identification research uses the CycleGAN [9] to increase
does not know whether or not the result of its classification the number of training samples and improve the accuracy
is correct during learning. The machine must find the rules of image recognition. Perez and Wang’s research (supple-
from all of the input examples in order to classify on its own. mentary source of literature) have also found that geometric

VOLUME 8, 2020 180751

 C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

transformation and brightness variation of image and

CycleGAN are able to enhance the image recognition
accuracy [10].

D. IDENTIFY THE LOCATION OF THE PEST WITH THE

TARGET DETECTION MODEL
With the advancement of technology and the development
of artificial intelligence, deep learning has significantly
increased the efficiency of image recognition, resulting in
many different artificial intelligence models that have encour-
aged the development in image recognition technology.
FIGURE 1. The flowchart of RNN stock price prediction.
These models can be divided into two types, the two-stage
object detector method and the one-stage object detector
method [11]. The recognition speed is faster, but the accuracy is lower.
Two popular one-stage object detector models are the Single
1) TWO-STAGE OBJECT DETECTOR METHOD Shot MultiBox Detector [14] and the YOLO [15] model.
The two-stage method is used to generate a region proposal The YOLO model only uses a single convolutional neural
after receiving the input image in order to identify the type of network (CNN) to determine the image inputs and outputs,
object. Its accuracy is higher, but it requires more computing so the efficacy of deep learning is greatly accelerated.
resources. The most representative model is the R-CNN [12] After inputting the image into the YOLO model, the fea-
and the Faster R-CNN [11], [13]. tures of the image are first extracted. Then, the image is
divided into an grid. The CNN is used to predict the number
1) The RNN is mainly used to solve time series problems.
of possible bounding boxes in each grid; the detected object
It typically has no temporal data in the input data.
type is calculated, and the image recognition result is then
In Fig.1, it can be seen that the RNN stores the output of
output.
the hidden layer in the memory, and when the next input
In the Titan X GPU hardware device environment, the
item has been entered, it also considers the last value
YOLO model can process 45 images per second, where the
stored in the memory in its calculations. For the simple
delay time for real-time video processing is less than 25 mil-
stock price learning RNN, we can input the Pt−1 data
liseconds.
to learn the Pt stock price, and M1 is the saved hidden
Since the YOLO model uses the entire image for training,
output. When we modify the Pt+1 value from training
the image detection accuracy will be higher with SSD.
Pt , M2 will be taken into consideration and added to the
The developer of the YOLO model continued to improve
output of Pt+1 .
the model and published the YOLOv3 model [16]. The
2) The Faster R-CNN method is divided into four parts:
YOLOv3 model uses multi-scale fusion for predictions and
a) Conv layers: The input image is reshaped and the increases the network architecture to 53 convolutional layers.
rescale factor is recorded in im_info as a proposal In Fig.2, The YOLOv3 model also introduces the concept of
layer for the ROIPooling alignment information. the residual network to increase the accuracy of the model
Then, the feature map is extracted after 13 Conv and improve the traditional YOLO model problem of having
layers, 13 ReLU layers, and 4 Max Pooling layers. poor recognition of small objects.
b) Region Proposal Network (RPN): The most sig- The YOLO algorithm extracts the features from an input
nificant difference from the Fast R-CNN is that image through the feature extraction network to obtain a
the RPN is used to generate region proposals. The fixed-size feature map, such as 13 × 13, and divides the input
RPN uses softmax to distinguish positive and neg- image into 13 × 13 grid cells. If the center coordinate of the
ative anchors, and the Bounding-box regression object falls in a grid cell in the ground truth, then the grid cell
is used to modify anchor position information to has predicted the object successfully.
obtain more accurate positions. The YOLOv3 model uses multiple-scale fusion methods
c) RoIPooling: The results from the feature maps to make predictions. It uses FPN-like up-sample and fusion
and the RPN are collected; the information is then methods (respectively 13 × 13, 26 × 26 and 52 × 52) to detect
integrated and sent to the FC for training. the multiple-scale feature maps. The detection outcome for
d) Classification: The ROIPooling input value is small targets is significantly improved. It is able to find the
used for classification and the Bounding-box location of the identified target in the entire image and frame
regression to obtain the best object position. it.

2) ONE-STAGE OBJECT DETECTOR METHODS E. ANALYSIS MODEL OF ENVIRONMENTAL SENSING DATA

One-stage object detector methods detect and identify In 2016, Liu et al. [12] proposed integration of the num-
objects simultaneously without a prior region proposal. ber of pests, risk level, and historical environmental data

180752 VOLUME 8, 2020

C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

FIGURE 3. The schematic diagram of the LSTM process.

FIGURE 2. The YOLOv3 network architecture diagram.

with the data provided by the user, where the time series
data set is based on the daily maximum temperature, daily
minimum temperature, daily maximum humidity, daily min-
imum humidity, and the number of pests caught per day.
The prediction model used in the network-based decision
support program is built on a mixed regression model, which
is referred to as the Vector Autoregression Moving-Average
with Exogenous Regressors (VARMAX) model with endoge-
nous and exogenous variables. The prediction results are
location-specific and can be used to determine the future risk
level. They can provide recommendations for pest prevention
and control so that farmers and pest managers can take pre-
ventative measures to avoid crop damage due to pests. These
functions can be used to implement the main components of
FIGURE 4. The mathematical representation of LSTM.
the IPM program, including the inspection, identification, and
control of pests in the orchard.
value, and multiplies the probability of the forget gate to
decide whether or not to forget the previously recorded value
F. DEEP LEARNING FOR DATA ANALYSIS
[c’ = g(z)f (zi ) + cf (zf )].
The LSTM (long short-term memory) model is currently
The value of ct−1 is a vector stored in the LSTM memory,
the most commonly used model in RNNs (Recurrent Neural
where the vector input at time t is zt . zt is multiplied by a
Networks) [17]. The flow chart of this model is shown in Fig.
linear conversion function to obtain g(zt ) as an input. zt is
3, which is composed of four main components: the input
multiplied by another linear conversion function to obtain
gate, the output gate, the memory cell, and the forget gate.
I (zt ) to control the input gate. zt is multiplied by another
1) Input Gate: When a feature is given as an input, the linear conversion function to obtain F(zt ) to control the forget
input gate controls whether or not to input the value in gate, and zt is multiplied by a linear conversion to get O(zt )
this iteration. to control the output gate. The LSTM flowchart is shown in
2) Memory Cell: Stores the calculated value to be used in Fig. 5.
the next stage. The value of g(zt ) is multiplied by the value of I (zt ) after
3) Output Gate: Controls whether or not to output the the Sigmoid function. The result is added to the product
estimated value in this iteration. of F(zt ) after the Sigmoid function and ct−1 and stored in
4) Forget Gate: Controls whether or not to clear the mem- the LSTM. The value of O(zt ) after the Sigmoid function
ory, similar to a restart. is multiplied by the stored value to obtain the output value
The mathematical structure of the LSTM model is shown at . The same steps are followed to calculate the value to be
in Fig. 4. The input is represented as g(z), and the input gate stored in the LSTM and the output value from the next input
is f (zi ); the general activation function f uses the sigmoid data zt+1 .
function to determine the probability of turning on the gate.
The probability that the input gate is turned on can be III. METHODS
determined by multiplying g(z) and f (zi ). The memory cell In this study, the pests are identified using artificial intelli-
records the current value of the input, adds the previous input gence; the pest recognition model is trained through deep

VOLUME 8, 2020 180753

 C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

FIGURE 6. The architecture of the proposed system.

FIGURE 5. The LSTM flowchart.

learning, and the image data for Tessaratoma papillosa are

collected using a mobile application and a drone from a
longan orchard. A real-time image recognition system for
Tessaratoma papillosa is established. It is combined with
environmental sensors to analyze the impact of environmental
factors on the pests’ life-cycle so that the system can promptly
inform the farmers regarding pest occurrences and damage, FIGURE 7. The labeled training data.

and the farmers can apply pesticides accurately on a timely

basis. The proposed system is able to provide improved con- were sent to the cloud database as well. In this research,
trol over Tessaratoma papillosa, reduce the labor costs in the approximately 687 images of adult Tessaratoma papillosa
orchard, and improve the yield and quality of the crop. were collected. The pest was first labeled in the image, which
The YOLOv3 model is used in this study, which is based is then augmented through image pre-processing, as shown in
on CNN and has good results in terms of image detection Fig. 7, and finally, the GPS information from the image was
and classification. The YOLOv3 model is used to classify used to indicate the location of the pest.
and label the pests. The drone and mobile application are
used to collect images of the pests in the orchards. Multiple B. IMAGE PRE-PROCESSING
environmental sensors are placed as meteorological observa- To improve the pest image recognition accuracy, we increased
tion stations for the orchard to regularly obtain environmental the number of training samples through image pre-
information including the temperature, humidity, and light processing, using feature enhancement, edge detection, and
intensity, for the analyses of the pest’s life-cycle and popu- grayscale processing.
lation. The purpose of the system is to predict the distribution
location and periods of occurrence of the pests, and to provide 1) COLOR FEATURES ENHANCEMENT
pest locations and damage level, as well as other relevant The color information was composed of red, green and blue
information to the farmers for real-time management of the (RGB) channels, and the color histogram records the occur-
orchard. rences of each color in the image formed three 256-dimension
We use drones and mobile devices to collect images of vectors. This vector was used to represent the color charac-
Tessaratoma papillosa, perform image enhancement and aug- teristics of the image.
mentation, and use the YOLOv3 model to classify and iden-
tify the processed image samples. We use a random gradient 2) SOBEL EDGE DETECTION
descent as the learning algorithm and mAP to assess the In the proposed method, the vertical Sobel edge detector
accuracy. The architecture of the proposed system is shown is first applied to enhance the vertical edges of the image,
in Fig. 6. followed by the application of the horizontal Sobel edge
detector to enhance the horizontal edges of the image. The
A. SAMPLE COLLECTION two images are then merged to obtain the final edge detected
We collaborated with the farmers to collect the training data image.
samples. The farmers used the mobile application that we
designed to take images of the pests. The mobile application 3) GRAYSCALE IMAGE PROCESSING
sent the pest identification results to the cloud database. Since the electromagnetic radiation (reflection or emis-
We also used a drone to collect images of the pest, which sion) of various substances causes them to have different

180754 VOLUME 8, 2020

C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

FIGURE 9. Tessaratoma papillosa images acquired from different viewing

directions.

FIGURE 8. Image pre-processing of samples to enhance the

characteristics of the pests. TABLE 1. 20 images per view to evaluate the recognition accuracy.

intensities, and photo-sensitive materials also have varied

sensitivities, the intensities of the detected radiation are inter-
preted as a grayscale image. The grayscale images are divided
into seven levels: white, gray-white, light gray, gray, dark the number of training samples and improve the recognition
gray, light black, and black, which serve as essential markers accuracy of the YOLOv3 model.
and the basis for image interpretation. In this work, we augmented the image data for the left,
Fig. 8 shows the different features of a captured image right, and abdominal views to increase the number of images
used as a multiple-input training data sample, where the in the training dataset.
separately obtained probability was accumulated through the In this study, six image augmentation methods were used to
deep learning model to improve identification accuracy. increase the number of images for the YOLOv3 training sam-
We analyzed the identification accuracy of Tessaratoma ples: normalization, left-right flip, edge sharpening, gamma
papillosa in different directions. For each of the back, left, contrast adjustment, Gaussian noise addition, and Gaussian
right, and abdominal perspectives, 20 pest images were used blur. Fig. 10 shows the image augmentation results.
to determine the recognition rate, as shown in Fig. 9. We used the 1) image augmentation method and 2) sample
Table 1 shows that the training data had significantly fewer compensation method to increase the number of left, right,
samples for the left, right, and abdominal views than for the and abdominal image samples before performing image aug-
back view, but the recognition rate for the back view was mentation for these images. Two different expanded sample
higher than 80%, and the recognition rates for the left, right, methods were used to evaluate the enhanced recognition
and abdominal views were less than 60%. accuracy of the pest images.
Many studies have found that data augmentation improves
model recognition accuracy, where it is recommended that 4) IMAGE AUGMENTATION METHOD
sufficient training samples be collected for image pre- Table 2 shows the pest identification accuracies for the back,
processing, such as cropping, rotation, contrast enhancement, left, right, and abdominal views after using six kinds of
noise addition, and edge sharpening. images augmentation processing: normalization, left-right
Imgaug is a library for Python with 98 types of image flip, edge sharpening, gamma contrast adjustment, Gaussian
augmenting functions that can be used to process images and noise addition, and Gaussian blur. It can be observed from
revise the image information. Table 2 that the recognition accuracy of each view was
After completing the labeling, the imgaug library was improved after image augmentation. However, the recogni-
used for image enhancement in machine learning and the tion accuracies of the left, right and abdominal views were
labeled information was automatically generated to increase still lower than that for the back view.

VOLUME 8, 2020 180755

 C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

TABLE 3. Accuracies after increasing the image samples followed by

image augmentation.

FIGURE 11. Convolutional layer and pooling layer.

Convolutional neural networks belong to one of the feed-

forward neural networks, which extracts image features and
completes image classification at the same time. The input of
FIGURE 10. The results of image augmentation. the neural network is a set of images, and each image passes
through several convolutional layers and places a pooling
TABLE 2. Accuracies after the equal-scale image augmentation method.
layer in the convolution layer appropriately, as shown in
Fig. 11. The features of the model are obtained first, and these
features are placed into the fully connected layer for clas-
sification. Image classification is achieved through repeated
learning in the deep layers.
The main differences between the CNN and the general
neural network are the CNN’s local perception and weight
sharing. The local features of the image are extracted by the
convolution kernel so that each area of the image shares this
5) SAMPLE COMPENSATION METHOD convolution kernel. This is a method widely used in the field
In this experiment, additional image data have the same of image recognition.
amount of sample images for the left, right, and abdominal The traditional CNN model was used in this study. After
views were generated by rotating the images at different inputting the original image data into the model, it only has
angles (90o, 180 o and 270 o), flipping the images left and to be classified in advance, and then it can be trained without
right, as well as through other operations. All of the images any labeling.
in the training samples underwent six types of image augmen- In this study, the number of pests was recorded to indicate
tation: normalization, left-right flip, edge sharpening, gamma the degree of damage and the location of the pests so the drone
contrast adjustment, Gaussian noise addition, and Gaussian could carry out accurate pesticide spraying. Therefore, this
blur. After the training, it was found that the recognition study used the YOLOv3 network architecture to re-train the
accuracy of each view increased to more than 90%. The target detection model.
results of the pest recognition accuracy are shown in Table 3. The YOLOv3 model is a Darknet-53 network structure that
After the SSD model was trained, the pest identification uses full convolution and introduces a residual structure. Due
accuracy reached 84%, and the pest classification accuracy to the large number of layers in the deep learning network,
reached 86%. gradient descent problems often occur during training. With
ResNet residual architecture, the YOLOv3 model reduces the
C. OBJECT DETECTION ON IMAGE CLASSIFICATION difficulty of training deep networks. The Darknet-53 network
Traditional fixed feature extraction methods often fail to system can be as deep as 53 layers and significantly improves
obtain useful features for image segmentation. In recent identification accuracy.
years, the convolutional neural network (CNN) architecture In the experiment, we calculated the average accuracies
was developed and has led to improved feature extraction. from 10 different angle images and found that when the

180756 VOLUME 8, 2020

C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

TABLE 4. The research results of the number of training samples that are
uneven with the YOLO V3 model, and the number of training samples is
average with the YOLO V3 model.

training samples were uneven (more images of the back,

but fewer of the left, right, and abdomen), the accuracy was
approximately 73%. We classified images of the back, left,
right, and abdominal angle images and then generated a lim-
ited number of training samples through data preprocessing FIGURE 12. The schematic diagram of the transmission mechanism of the
wireless sensing platform.
(rotation and cropping) so that all angles had the same number
of training samples. The accuracy of pest recognition rose to
more than 92% after training using the neural network with
an average amount of training samples.
We compared the research results for an uneven number of
training samples with an average number of training samples
using the YOLOv3 model, as shown in Table 5. Based on
the results, we inferred that a useful data pre-processing
method can improve the efficiency of deep learning as well
as recognition accuracy.

D. SENSOR DISTRIBUTION
Crop pests thrive in excessively dry, hot environments. In this
FIGURE 13. The actual configuration of the sensors in the field.
research, we combined environmental sensors through a wire-
less environmental sensor transmission platform and a cloud
big data computing server system to collect environmental used to measure six kinds of environmental data, including
data. These data were analyzed to determine the real-time temperature, humidity, light, soil humidity, atmospheric pres-
agricultural meteorology, soil conditions, crop growth, and sure, and altitude.
other relevant information necessary to assist the manage- We use the DS3231 module to calibrate the time on the
ment of an orchard environment. microprocessor. The Raspberry Pi receives the environmental
The research and development of wireless sensor transmis- data from the sensors every hour. It transfers the data to the
sion systems focus on multiple sensors to transmit services Cloud via a wireless network and stores the data in the SD
through a wireless network in real-time and send the data module on the client-side at the same time to avoid data loss
back to the system for analysis. It is one of the essential through failed wireless transmission.
development applications of the IoT in smart agricultural The environmental data are transmitted to the Raspberry
technology. Pi on the server station from the client-side through the
In this study, the sensor modules were evenly arranged in SX1276 Lora module via a wireless network. The Raspberry
an orchard on a hillside according to the wireless transmission Pi uploads the data to the cloud database to provide devel-
sensor characteristics. The environmental data collected by opers with real-time observations of environmental changes,
the sensor modules were aggregated and transmitted to the as well as data retrieval and analysis.
cloud database through a wireless communication protocol To collect the environmental data in the field any time we
for subsequent analysis and calculation. wanted, we set up six sets of sensors and a LORA trans-
Fig. 12. shows a schematic diagram of the transmission mission module on the client-side. The LORA module is
mechanism of the wireless sensing platform in the orchard used to transmit the data to the Raspberry Pi on the server
on a slope. station, which transfers the data to the cloud database through
We obtained 6 evenly distributed sets of environmen- a wireless network to provide query and analysis functions
tal sensing modules in the orchard, as shown in Fig. 13, to the managers in real-time. The architecture is shown in
to observe the proliferation of pests in different environments. Fig. 15.
Fig. 14 shows the designed environmental sensing module. The six sets of client sensors send data to the server sta-
The Arduino Nano was used in a small embedded platform. tion at different times. To ensure the correctness of the data
The four sensors, GY-30, soil, DHT22, and BMP180, were received by the server station, the server station returns an

VOLUME 8, 2020 180757

 C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

FIGURE 16. (a) The external module of the environmental sensors (b) The
FIGURE 14. The hardware architecture of the sensor box content. internal module of the environmental sensors.

The proposed system integrated an agricultural manage-

ment mobile application to provide pest image collection,
real-time pest identification, and visualization of sensor data.
The developed mobile application provides real-time access
to environmental data and displays the historical records of
the environmental sensors. In this work, the analytical results
from the environmental data are displayed in the form of an
App on the mobile device to provide farmers with real-time
agricultural management. The mobile application is divided
into two parts and is described follows:

1) BROWSING OF THE ENVIRONMENTAL SENSORS

FIGURE 15. Transmission architecture of client-side sensor and server DATABASE
station sensor.
The environmental sensors deployed in the orchard allow
users to monitor the current growing environment of the
acknowledgment value to the client-side after receiving the fruits trees. The historical environmental data also provide
data. Otherwise, the client re-sends the data. information about the environmental conditions.
The environmental sensing module in this experiment
included the Lora transmission module, sensors for tem- 2) UAV MISSION DISPATCH MODE
perature, humidity, soil humidity and atmospheric pressure, Users can use drones to plan precise pesticide spraying oper-
a clock module, an SD card module, a micro-controller ations in real-time, either using manually-controlled flights
Arduino Nano, and a lithium battery. All components are or pre-input flight paths obtained through the mobile appli-
shown in Fig.16. cation.
In this work, multiple sensors are placed in an orchard to
E. ANALYZING ENVIRONMENTAL SENSOR DATA WITH collect environmental data and provide instantaneous infor-
LSTM mation on pest damage in various areas of the orchard, so that
We collected a total of 6 sets of environmental sensor infor- the correlation between pest occurrence and related environ-
mation on the slope area in a longan orchard in Nanhua mental factors can be determined. We used the LSTM in the
District in Tainan, Taiwan, from January to June 2020. Data RNN neural network to build a suitable training model based
for each environmental sensor module was collected at one on an on-site investigation of the pest severity in orchards by
hour intervals, where each set of data included real-time professional farm managers and the environmental data. The
environmental information for the time, temperature, humid- model is able to analyze the predicted data and determine
ity, light, soil humidity, atmospheric pressure, and relative the extent of pest occurrences. The prevention and control
altitude. Up to the present time, we have collected a total of information are then provided to the farmers as quickly as
12,960 sets of environmental data. The environmental data possible to reduce the agricultural damage caused by the
had slight variations due to the location of the sensor module. identified pests.
Hence, we used LSTM to analyze the six sensor modules
to predict environmental factors at six locations and used IV. RESULTS
machine learning methods to predict whether environmental In the experiment, the data was processed, and image aug-
factors will result in the presence of pests. mentation was performed to increase the number of training

180758 VOLUME 8, 2020

C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

FIGURE 18. The schematic diagram of the user interface.

FIGURE 17. The experiment results of using K-means to find a suitable

anchor box.

samples. The image augmentation methods used in this work

included left and right reversal, angle conversion, Gaussian
blur, and zoom in and out, among others. The number of
training samples was increased from 849 to more than 7,000.
The YOLOv3 model was used for deep learning training. The
anchor box in the YOLOv3TensorFlow model was obtained
by clustering the VOC data set. However, the 20 types in the
VOC data set comprise large differences in the target object
FIGURE 19. The spatial distribution of identified pests and the impact
sizes, ranging from large objects such as a bicycle or a bus to map.
smaller objects such as a bird or a cat. Therefore, the VOC
data set was not suitable as a data set for target training and TABLE 5. The correlation adjustment between the score/IOU parameters
detection in this experiment. In the experiment, we found the and mAP.

anchor box with k = 9 to be more suitable for the target

detection in this work, as shown in Fig. 17.
The Stochastic Gradient Descent is used in this work as
the learning algorithm. The gradient descent algorithm uses
each weight update and parameter deviation to advance each
update to a negative gradient in order to obtain a minimum
error function value. The formula is as follows:
is assigned a value from 0 to 1. The experimental results are
θl+1 = θl − a∇E(θl ) (1) shown in Table. 4.
During the training process, the YOLOv3 model continu-
where l is the number of iterations; a is the learning rate, ously uses different parameters in an attempt to find the most
which is set at 0.0001 in this experiment; θ is the parameter suitable model for the given dataset. The test data is given as
vector, and E(θ) is the loss function. input into the model and combined with the sensor data from
The gradient ∇E(θ) of the loss function is calculated by the orchard to analyze the level of damage and the growth
taking a subset of the training data to measure and update the stages of the pest. It has been found that the mAP accuracy is
gradient parameters. The process for updating and adjusting 0.92.
the gradient of the entire training data set is called an epoch. To make the system more accessible and convenient to
In object recognition, the score and IOU are significant farmers, we designed a mobile application with a photog-
figures. The score is given by the target frame when the model raphy function. The mobile application is also capable of
recognizes it as having the highest pest probability. The IOU updating the environmental data at any time. The mobile
is the intersection between the two rectangles representing application is shown in Fig. 18 and Fig. 19. The mobile appli-
the frame of the recognition result and the correct target. cation makes it possible for farmers to identify pests while
The amount of intersection between the rectangular frames patrolling the orchard. The identification results are sent to

VOLUME 8, 2020 180759

 C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

the cloud to analyze the location of the pest, so the farmers above and continue to experiment to find solutions, hoping
can be informed as to the best time for pesticide spraying with to achieve immediate recognition of pest by drones as soon
the goal of improving pest prevention and management. as possible and establish intelligent agriculture.

V. CONCLUSION REFERENCES
It is difficult to obtain image data of pests since they prefer [1] L. M. Gladence et al., ‘‘Recommender system for home automation using
hiding behind leaves and in the tops of trees. Therefore, IoT and artificial intelligence,’’ J. Ambient Intell. Hum. Comput., 2020, doi:
10.1007/s12652-020-01968-2.
we applied image data augmentation to increase the number [2] S. S. Priya, L. M. Gladance, A. Valarmathi, and J. M. S. Joy, ‘‘Brain tumor
of pest training samples and improve the recognition accuracy detection and disease prediction using CNN algorithm,’’ in Test Engi-
for Tessaratoma papillosa. neering and Management, vol. 82. Oakland, CA, USA: The Mattingley
Publishing Co., Inc., Jan./Feb. 2020, pp. 13376–13381.
In this study, the YOLO series (You Only Look Once, [3] G. L. Mary, C. K. Vakula, M. P. Selvan, and T. Y. S. Sam-Hita, ‘‘A research
YOLO) neural network algorithm was used to identify the on application of human-robot interaction using artificial intelligence,’’ Int.
pests in the input images. The recognition rate was about J. Innov. Technol. Exploring Eng., vol. 8, no. 9, pp. 784–787, 2019.
[4] V. Martin and S. Moisan, ‘‘Early pest detection in Greenhouses,’’ in Proc.
90%. The main target was Tessaratoma papillosa. Drones Int. Conf. Pattern Recognit., 2008, pp. 1–4.
were used to collect images of these pests on fruit trees in [5] K. Wang, S. Zhang, Z. Wang, Z. Liu, and F. Yang, ‘‘Mobile smart device-
an orchard on a slope. A mobile application was used to based vegetable disease and insect pest recognition method,’’ Intell. Autom.
Soft Comput., vol. 19, no. 3, pp. 263–273, Aug. 2013.
collect pest images from the ground at the same time. The [6] J. L. Miranda, B. D. Gerardo, and B. T. Tanguilig, III, ‘‘Pest detection and
pest images were uploaded to the Cloud for image recog- extraction using image processing techniques,’’ Int. J. Comput. Commun.
nition in real-time, and the pest location and distribution Eng., vol. 3, no. 3, pp. 189–192, 2014.
[7] M. D. Gondal and Y. N. Khan, ‘‘Early pest detection from crop using image
were determined for the purpose of pesticide application. The processing and computational intelligence,’’ FAST-NU Res. J., vol. 1,
proposed system is designed to facilitate farmers in advanced no. 1, pp. 59–68, Jan. 2015.
pest control operations. [8] W. Ding and G. Taylor, ‘‘Automatic moth detection from trap images
for pest management,’’ Comput. Electron. Agricult., vol. 123, pp. 17–28,
In this study, a deep-learning image recognition method for Apr. 2016.
Tessaratoma papillosa was proposed that uses the YOLOv3 [9] A. Almahairi, S. Rajeswar, A. Sordoni, P. Bachman, and A. Courville,
model to mark and classify the pests in images and extract the ‘‘Augmented CycleGAN: Learning Many-to-Many mappings from
unpaired data,’’ 2018, arXiv:1802.10151. [Online]. Available:
characteristics of the pests. The pest identification accuracy http://arxiv.org/abs/1802.10151
and model training time are used to determine the most [10] C. Cortes and V. Vapnik, ‘‘Support-vector networks,’’ Mach. Learn.,
suitable model for the application. Also, the data from the vol. 20, no. 3, pp. 273–297, 1995.
[11] F. Faith praise, P. Birch, R. Young, J. Obu, B. Faith praise, and C. Chatwin,
environmental sensors was used to analyze locations prone ‘‘Automatic plant pest detection and recognition using k-means clustering
to the presence of pests. The location and distribution of the algorithm and correspondence filters,’’ Int. J. Adv. Biotechnol. Res., vol. 4,
pests were instantly provided to farmers to provide accurate no. 2, pp. 189–199, 2013.
[12] Z. Liu, J. Gao, G. Yang, H. Zhang, and Y. He, ‘‘Localization and classi-
pesticide application, reduce agricultural pest damage, and fication of paddy field pests using a saliency map and deep convolutional
increase crop quality and yield. neural network,’’ Sci. Rep., vol. 6, no. 1, Apr. 2016.
Since Tessaratoma papillosa mostly grows on the backs [13] S. J. Pan and Q. Yang, ‘‘A survey on transfer learning,’’ IEEE Trans. Knowl.
Data Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2010.
of leaves and in treetops, it is difficult to collect training [14] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ‘‘ImageNet classification
samples for the purpose of identifying these pests. Further, with deep convolutional neural networks,’’ in Proc. Adv. Neural Inf. Pro-
the inconsistency of outdoor lighting results in low image cess. Syst., 2012, pp. 1097–1105.
[15] K. Simonyan and A. Zisserman, ‘‘Very deep convolutional networks for
recognition accuracy. Therefore, here, an APP installed on large-scale image recognition,’’ 2014, arXiv:1409.1556. [Online]. Avail-
a mobile phone was used to collect images of Tessaratoma able: http://arxiv.org/abs/1409.1556
papillosa that appeared on the back of the leaves in order [16] C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY,
USA: Springer, 2006.
to locate these pests. We plan to apply the Tessaratoma [17] D.-R. Liu, S.-P. Chuang, and H.-Y. Lee, ‘‘Attention-based memory selec-
papillosa identification model to small drones in the future, tion recurrent network for language modeling,’’ 2016, arXiv:1611.08656.
hoping to reduce the human effort for collecting pest image [Online]. Available: http://arxiv.org/abs/1611.08656

samples by the maneuverability of drones. It is hope that this

will increase the number of images of Tessaratoma papillosa
that appear in treetops and improve identification accuracy. CHING-JU CHEN received the Ph.D. degree in
Some difficulties were encountered in the research insti- engineering science from National Cheng Kung
tute: The first was the need to continuously increase the University, Taiwan, in 2011. She is currently
different angles in the images to solve the issue of insufficient an Assistant Professor with the Department of
Bachelor Program in Interdisciplinary Studies,
training samples; the second was to stabilize the drone flying National Yunlin University of Science and Tech-
between the trees to get good quality images of pests and nology, Taiwan. Her research interests include
resolve the disturbances of the leaves caused by the wind artificial intelligence, UAV system integration,
from the UAV propeller; the third was speeding up the image embedded systems, environmental resource appli-
cations, computational intelligence applications
recognition process to improve pest positioning efficiency to UAV telemetry image processing, geographical information systems,
and reduce the time required for farmers to inspect the area global positioning systems, remote sensing, computer vision, and image
for pests. We plan to overcome the problems mentioned recognition.

180760 VOLUME 8, 2020

C.-J. Chen et al.: AIoT Based Smart Agricultural System for Pests Detection

YA-YU HUANG was born in Changhua, Taiwan, CHUAN-YU CHANG (Senior Member, IEEE)
in 1997. She received the B.S. degree in com- received the Ph.D. degree in electrical engineering
puter science and information engineering from from National Cheng Kung University, Taiwan,
the National University of Tainan, Tainan, in 2019. in 2000.
She is currently pursuing the master’s degree with He was the Chair with the Department of
the Department of Engineering Science, National Computer Science and Information Engineering,
Cheng Kung University. From 2018 to 2019, she National Yunlin University of Science and Tech-
was a Research Assistant with the Knowledge, nology (YunTech), Taiwan, from 2009 to 2011.
Information and Database System Laboratory. She From 2011 to 2019, he served as the Dean of
is also working on machine learning applied to Research and Development and the Director of
computer vision and image processing in robotics applications. Her research the Incubation Center for Academia-Industry Collaboration and Intellec-
interests include remote sensing applications in agriculture, applications of tual Property, YunTech. He is currently the Deputy General Director of
machine learning in satellite imagery, optical satellite and aerial imagery, and the Service Systems Technology Center, Industrial Technology Research
image processing. Institute (ITRI), Taiwan. He is also a Distinguished Professor with the
Department of Computer Science and Information Engineering, YunTech.
His current research interests include computational intelligence and their
applications to medical image processing, automated optical inspection,
emotion recognition, and pattern recognition. In the above areas, he has more
than 200 publications in journals and conference proceedings.
Dr. Chang is a Fellow of IET and a Life Member of IPPR and TAAI.
He served as the Program Co-Chair of TAAI 2007, CVGIP 2009, 2010–2019
International Workshop on Intelligent Sensors and Smart Environments, and
the Third International Conference on Robot, Vision and Signal Processing
(RVSP 2015). He served as the General Co-Chair of 2012 International
Conference on Information Security and Intelligent Control, 2011–2013
Workshop on Digital Life Technologies, CVGIP 2017, WIC2018, ICS 2018,
and WIC2019. From 2015 to 2017, he was the Chair of the IEEE Signal Pro-
cessing Society Tainan Chapter and the Representative of the Region 10 of
IEEE SPS Chapters Committee. He is also the President of the Taiwan
Association for Web Intelligence Consortium.

YUEH-MIN HUANG (Senior Member, IEEE)

received the M.S. and Ph.D. degrees in electri-
cal engineering from The University of Arizona,
in 1988 and 1991, respectively.
He is currently the Chair Professor with the
Department of Engineering Science and the Insti-
tute of Education, National Cheng Kung Uni-
versity, Taiwan. He has trained over 60 Ph.D.
and 300 M.S. thesis students. He has coauthored
three books and has published more than 280 ref-
ereed journal research articles. His research interests include e-Learning,
YUAN-SHUO LI was born in Tainan, Taiwan, multimedia communications, and artificial intelligence. He became a Fel-
in 1996. He received the B.S. degree in com- low of the British Computer Society, in 2011. He received many research
puter science from National Pingtung University, awards, such as the Taiwan’s National Outstanding Research Award, in 2011
Pingtung, in 2018. He is currently pursuing the and 2014, and the 2017 Taiwan Outstanding IT Elite Award. He is in the
master’s degree with the Department of Engi- editorial board of several international journals in the area of educational
neering Science, National Cheng Kung Univer- technology, computer communications, and web intelligence. He is also the
sity, Tainan. His research interests include remote Founding Chair of International Symposium of Emerging Technologies for
sensing imagery in multispectral, application of Education (SETE) and International Conference of Innovative Technologies
embedded systems, and LoRa wireless transmis- and Learning (ICITL).
sion technology.

VOLUME 8, 2020 180761