A Review of Application of Deep Learning for

Weeds and Crops Classification in Agriculture

Syed I. Moazzam Umar Shahbaz Khan Mohsin Islam Tiwana
Department of Mechatronics Department of Mechatronics Department of Mechatronics
National Centre of Robotics and National Centre of Robotics and National Centre of Robotics and
Automation Automation Automation
National University of Sciences and National University of Sciences and National University of Sciences and
Technology Technology Technology
Islamabad, Pakistan Islamabad, Pakistan Islamabad, Pakistan
[email protected] [email protected] [email protected]

Javed Iqbal Waqar S. Qureshi Syed Irfan Shah

National Centre of Robotics and Department of Mechatronics Department of Mechatronics
Automation National Centre of Robotics and National University of Sciences and
National University of Sciences and Automation Technology
Technology National University of Sciences and Islamabad, Pakistan
Islamabad, Pakistan Technology [email protected]
[email protected] Islamabad, Pakistan
[email protected]

Abstract—Weeds are major cause due to which farmers get Manual weeding is difficult and labor costing that’s why
poor harvest of crops. Many algorithms are developed to classify making it unfeasible. Mechanical weeding is comparably
weeds from crops to autonomously destroy weeds. Color-based, efficient but could not remove intra-row weeds and
threshold-based and learning-based techniques are deployed in sometimes it could damage the main crop due to human error.
the past. From all techniques, deep-learning-based techniques
Herbicide usage is most common due to its high efficiency in
stand out by showing the best performances. In this paper, deep-
learning-based techniques are reviewed in the case where these controlling weeds [7]. The use of herbicides in the complete
are applied for weed detection in agricultural crops. Sunflower, agricultural field causes waste of herbicide and
carrot, soybean, sugar beet and maize are reviewed with respect environmental pollution [8]. Using half dosage in low-
to the weeds present in them. Deep learning structures and density weed area can produce the same result as full dosage
parameters are presented, and research Gaps are identified for in low-density weed area [9]. Chemical weeding is at number
further research. one as a technique used against weeds. In Pakistan it is done
two ways, one manually sprayed by farmer, which have
Keywords—Convolutional Neural Networks, Deep learning, detrimental effects on farmer health and second, uniformly
Image processing, Precision agriculture, Smart farming, Weed
detection.
sprayed using tractor and assisting setup which result in high
cost of herbicide and unwanted environmental pollution.
I. INTRODUCTION To avoid pollution and to minimize herbicide loss, herbicide
It is estimated that the world population will reach around spray must be done using modern technology autonomously.
9 billion by 2050 and to meet the food demand agricultural The first step in this regard is to classify weed, crop, and soil
food production must be doubled [1], [2]. On the other hand, separately and efficiently.
weeds along with some other problems and challenges are the Many image processing approaches have been attempted
main causes which are faced by agriculture [3]. Many studies so far. There are three major types of approaches to handle this
have been done to control weed, which grows in the field and classification between weeds and crops in the past. Color-
share everything needed by the main crop, as a result, main based classification, threshold-based classification, and
crop losses targeted yield [4], [5]. Weed compete with crop in learning-based classification. Color and threshold-based
the domain of space soil moisture, nutrients and solar radiation approaches suffers loss in accuracy when the light is too high
[6]. Weeds should be removed at every step of the growth of or too low, learning-based approaches promise more precision
any crop especially at the initial stages to reduce overall on the other hand. From learning-based approaches, two best
production loss. Weeds have a negative effect on crops which approaches that stand out due to high accuracy are support
include sharing of water, light, nutrients and space, increased vector machines and deep learning. Deep learning has placed
production costs, difficulty in harvesting, depreciation of itself in the first position by delivering maximum accuracy
product quality, increased risk of pests and diseases and compared to all other techniques. In this review paper, deep-
decrease in commercial value of cultivated areas. Weeds are learning-based approaches are discussed, which are deployed
the most challenging problem of farmers as these threats their previously in the topics of weed detection or weed and crop
ability to produce good quality food in a cost-effective way. classification.
There is a strong link between weed competition and crop
II. DEEP LEARNING
yield loss.
Deep learning is a branch of machine learning. There are
three types of learning i.e. supervised, semi-supervised and

978-1-7281-3058-3/19/$31.00 © 2019 IEEE

unsupervised. Deep learning architectures could be deep TABLE I
neural networks, deep belief network, recurrent neural Different Deep learning layers
networks, and convolutional neural networks. Deep learning Layer Discription
uses a cascaded structure of layers to extract features and does Image input It inputs 2-D images to a network and normalize
classification or pattern analysis. A deep neural network is a layer data.
Convolution Convolutional filters are applied to the input.
type of artificial neural network which has many layers layer
between input and output. The network calculates the Rectified Performs a threshold operation to each element
probability of each possible output at all layers. Iterations are linear unit Values less than zero is set to zero.
done, and the network is trained to get features out of an (ReLU) layer
image, the network finds the trends for all classes which are Leaky ReLU Performs a threshold operation
needed to be classified, once trained, a network could classify layer Values less than zero is multiplied by a fixed
scalar.
new input images automatically. Convolutional neural
Tanh layer Applies the tan hyperbolic activation function
networks are known to be the best classifiers when the input Average Performs down-sampling
type is images. pooling layer Input is divided into rectangular pooling regions
and average values are computed
Deep learning has advantage over all other machine Max Pooling Performs down-sampling
learning-based algorithm that it can learn features on its own layer Input is divided into rectangular pooling regions
that's why eliminating need to compute and design features and maximum values are computed
manually which save time also for other techniques features Fully Multiplies the input by a weight matrix and then
tend to change if problem or dataset is altered. Another Connected adds a bias vector.
advantage of deep learning is its unlimited accuracy, better layers
SoftMax Applies a SoftMax function
training or more data input could lead to better accuracy than layer
the previous one. Classification Computes the cross-entropy loss for multi-class
layer classification problems with mutually exclusive
The disadvantage of deep learning is longer training time classes.
but if time taken to create features for other learning-based
techniques is considered then deep learning time becomes papers are reviewed which provide a variety of deep learning
negligible. The testing time is less compared to other machine application for the reader. These short-listed papers are
learning-based methods. For best performance, deep learning enough to provide deep learning trend for a new researcher in
needs a large input dataset. Dataset's problems could be low this field. All the selected papers are described one by one and
variation between classes, noise, low resolution, crop at the end comparison is drawn between them.
occlusions, and sensors inaccuracy. Deep Learning is
computationally expensive. Deep learning typical structure is IV. LITERATURE REVIEW
shown in Fig. 1.
A. Sunflower and Weed Classification with deep learning
Input passes through several blocks composed of
Fawakherji et al. [10] did crop and weeds classification
convolution, Rectified linear units (ReLU) and pooling layers
and in this process, the algorithm learns features. After feature by pixel-wise segmentation using a convolutional neural
learning classification is done by simplifying matrices, network. They proposed three steps algorithm to perform
passing vectored input from several fully connected layers and classification between crop and weeds, the first step is
then SoftMax function is applied and finally classification in segmentation in which semantic segmentation UNet is
done. Different Deep learning layers are given in Table I. applied to classify between vegetation and soil pixel by pixel.
The applied net has a coder-decoder based structure.
III. METHODOLOGY Modified VGG-16 structure was used in this step. Second
The papers selected for this review have two must step was blob extraction in which images region of interest
conditions, one is deep learning application and the other is were extracted from the previous step. Dilation is applied on
weed detection or classification. The papers are searched from the binary mask obtained in the previous step and connected
Science Direct, IEEE, Taylor and Francis, and Elsevier with blobs are chosen and their bounding box is defined. And
keywords of "deep learning" and "weed detection". Thirty finally, corresponding RGB patches are cropped using these
conference and Journal papers were downloaded initially, bounding boxes from original RGB images in this step. The
after that only recent publications from 2017 to 2019 are third and last major step is the classification of crop/weeds
sorted out, which were 14, which show current deep learning using CNN. Cropped patched from the previous step are fed
trend applied to agricultural crops. After that those to a VGG-16 model. This model contains 13 convolutional
publications with less than 5 pages or publications already layers with 3×3 kernel size, maximum pooling kernel, and
cited in most recent papers are filtered out. In the end, seven stride of 2 for down-sampling.

CLASSIFICATION MODULE
FEATURE DETECTION MODULE

CONVOLUTION POOLING CONV. POOLING OUTPUT

INPUT + + CLASS
IMAGES RELU RELU

FLATTEN FULLY SOFTMAX

CONNECTED

Fig. 1. Deep learning typical structure

Dataset contains sunflower images from Italy, images were percent for carrot plant and 88.1 percent for weeds.
taken in spring 2016. For training and classification NVIDIA McCool et al. [12] adapted Inception-v3 model, used it to
GTX 1070 GPU is deployed with VGG-16 architecture. train a smaller deep convolutional neural network and then
Dataset contained 500 sunflower images which were they mixed few lightweight models to enhance their
increased to 2000 by data augmentation techniques using performance. First, they picked up Google Net i.e. Inception-
rotations, zooming and flipping both ways, of which 1500 v3, trained on ImageNet dataset containing 1000 classes. In
images are used for training, 350 for validation and 150 the second step, they trained two lightweight DCNN with a
images for test. Testing results were 90 percent accurate for low number of parameters. These lightweight models were
both classes of weeds and sunflowers. obtained by compressing the Inception-v3.
Finally, in the third step different lightweight models were
B. Carrot and weeds Classification with deep learning
combined to hope for better accuracy and greater speed. They
Knoll et al. [11] captured RGB images and converted used a carrot dataset in which there were 20 training and 40
them to HSV format. The preprocessing of images can be testing crop/weed images. Inception-v3 model provided 93.9
better understood by block diagram given by authors as percent accuracy at the computational complexity of 25
shown in million parameters, so the model gave best accuracy but at
Fig. 2. higher complexity and training time while mixture of eight
small lightweight networks provided accuracy of 90.5% with
network complexity of just 2 million parameters, so in this
technique, trade-off exists between accuracy against network
complexity and speed. This type of trade-off is important to
consider where heavy networks could not perform on real-
time applications.
Fig. 2. Preprocessing of images [11]
C. Soybean and weeds Classification with deep learning
Hue and saturation images are further processed to Tang et al. [13] used K-means feature learning
obtain a mask which is used to extract green content from accompanied by CNN to classify between soybean and some
background. Bandpass filter is applied on the hue channel to weeds. Image acquisition was done in the north campus of
compute H-bin and Saturation image is thresholded at 16 northwest Agriculture and Forestry University with Canon
percent of fall of curve in histogram and in this way S-bin is EOS 70D camera in June and July months of 2016. Camera
extracted, mask is generated by AND operation between H height was kept at 40 to 69 centimeters above the plants.
and S bins. When this mask is multiplied by the original RGB Weeds targeted included Cephalanoplos, Digitaria and
image, background free vegetation image is achieved which bindweed. Dataset was captured in different light and soil
is used in further processing. conditions. Authors used k-mean training prior to CNN
CNN used for training in this problem obtained 101 ×101 training as shown in the block diagram of their algorithm as
pixels input images and classify weed and carrot or shown in Fig. 3.
background i.e. three output classes. The network used One thing not shown in their block diagram is ReLU
contains eleven layers, layers properties are listed in Table II activation function, which they had used before every down-
[11]. sampling layer. Input layer dimensions are 265×256×3.
For training this CNN classifier on 500 images, CUDA is
deployed on GTX Titan having 6GB graphic memory. This
network was trained for three weeks. Images were taken with
Nikon D5300 camera. The accuracy of recognition was 98
TABLE II
Eleven Layers of Network
Layers Operation No of kernel Activation
filters size
1 Input image
RGB 101×101
2 Convolution 16 5×5 Leaky
layer Rectify
3 Convolution 32 7×7 Leaky
layer Rectify
4 Pooling layer 2×2,
stride = 2
5 Convolution 32 5×5 Leaky
layer Rectify
6 Convolution 64 7×7 Leaky
layer Rectify
7 Pooling layer 2×2,
stride = 2
8 Dense layer 64 Tanh
9 Dense layer 64 Tanh
10 Dense layer 64 Tanh
11 Output layer 3 Fig. 3. Tang Algorithm [13]
 They obtained 256 features of size 8×8×3 by k-means pre- The maize and associated weeds dataset was captured in
training. These features are used as convolution filters in the its initial stages in Pillaro city of Tungurahua province in
first convolution layer. They used stride of 4 for first Ecuador. Images were normalized in green channel to
convolution and max-pooling is used as a down-sampling improve green color detection; greyscale images were
procedure with stride of 2×2. obtained equivalent to green color and OTSU method was
The total image dataset counted to 820 RGB images of deployed to get binary mask of images. Using obtained binary
size 256×256 of which Soybean and Cephalanoplos had 210 mask, images were segmented and saved in PNG format and
samples each and Digitaria and bindweed contained 200 labeled manually. For training 2835 maize and 880 weed
samples each. For training learning rate of 0.01 was selected images were taken and rotated every 30 degrees to get
and 40 samples of each class were selected, and the rest of augmented images thus increasing dataset by 12 times, 5th
the samples were used in testing in contrast to normal practice part of this dataset was reserved for validation.
where training data is always kept large. Using k-mean with LeNet, AlexNet, SNET, and CNET are four
CNN provided 92.89 percent test accuracy as compared to Convolutional Neural Networks (CNN) which were deployed
91.07 percent which was achieved using random for training among which CNET performed the best. Image
initialization. input size was 64x64 pixels, the first convolution layer was
applied with 16 filters,5 kernel size and stride of 1 pixel, after
D. Sugar beet and weeds Classification with deep learning that max-pooling was applied with kernel size and stride of
Milioto et al. [14] deployed CNN based semantic 2. Then second convolution layer used the same parameters
segmentation to differentiate between sugar beet and weeds. as 1st one following a new max pooling layer with the same
He deployed an encoder-decoder based architecture for this parameters as the previous pooling layer. After that resulting
purpose. Fourteen images are fed as input to the network, image was obtained of the size of 16x16x16. After that, three
these fourteen images description is given by authors as fully connected layers were applied with 384, 192 and 2
shown in Table III. neurons respectively. Training accuracy of 97.26 percent is
Input to the network were 512×384 pixel images. All reported.
convolutional layers were followed by batch normalization in For testing 404 images were used for which half were weeds.
which ReLU was applied. Testing accuracy was reported 92.08 percent for weeds and
89.11 percent for maize. The testing speed was checked for
The images data was captured with four-channel (RGB+NIR)
GPU, CPU and Raspberry Pi and was reported to be
JAI AD-130 GE camera but only RGB images were used in
58.47,5.08 and 0.36 FPS respectively.
this study. Near 10,000 images were utilized with 70, 15 and
Chavan et al. [16] created AgroAVNET which is a hybrid
15 percent for training, validation and testing purposes
model of AlexNet and VGGNet for weed and crop
respectively. On NVIDIA GTX1080Ti training was done
recognition. It has five convolution layers with filter sizes of
with 200 epochs in about 48 hours. Testing accuracy is
11×11, 5×5 and 3×3 in first, second and other three
reported 94.74 percent on the same reserved 15 percent
convolution layers respectively and the number of filters in
dataset. The authors used two other completely different
these five layers were 96, 256, 384, 384 and 256. Three fully
datasets and the testing accuracy was 72 and 77 percent in
connected (FC) layers were applied with 4096 neurons in first
those datasets.
two FC layers and n nodes equal to output classes in the last
E. Maize and Weed Classification with deep learning FC layer. Relu is applied for rectification and SoftMax
Andrea et al. [15] applied convolutional neural networks function is used in the last layer for classification.
for classification between maize and weeds. Image VGG16 network contains 16 layers in total of which 13
acquisition was done with Raspberry Pi 3 with Pi camera are convolution layers, number of filters in the first two layers
version 2.1 and video resolution was set to 1280x720. Core are 64, 128 in the next two layers, 256 in next three layers and
i7 2.7 GHz 8 core CPU Computer with Nvidia GTX950M 512 in all other convolutional layer, all filters have 3×3 size.
graphics card was used for training network and Raspberry Pi The network has five max-pooling layers. Rectification in the
3 with ARM Cortex-A53, 1.2 GHz containing 4 cores was last layer is done with SoftMax function, all other layers
used to test the network. OpenCV and Caffe were used as a applied Relu function.
processing software on Ubuntu. The hybrid AgroAVNET takes normalization of AlexNet
and depth of filters of VGGNet, the architecture is shown in
TABLE III
Fig. 4.
Input Channels of Milioto Algorithm [14] Plant seedlings dataset was used which has 5544 images
1 Image R Component 8 Hue component (from HSV) of 12 classes as shown in Table V, of which 4750 were used
2 Image G Component 9 Saturation component (from for training with 10 fold cross-validation. For processing,
HSV) system specifications were Intel Xeon E5-2695, 64GB RAM
3 Image B Component 10 Value component (from
HSV)
and NVIDIA TITAN XP with 12GB RAM, python with
4 Excessive Green 11 Sobel in x direction of Keras library was used as software. Test accuracy of
Excessive Green 93.64±0.81 is reported with AgroAVNET. Summary of
5 Excessive Red 12 Sobel in y direction of reviewed papers is tabulated in Table IV.
Excessive Green
6 Color Index of 13 Laplacian on Excessive V. RESEARCH GAPS
Vegetation Extraction Green
7 Normalized Difference 14 Anny Edge Detector on Deep Learning consumes big data to deliver higher
Index Image Excessive Green accuracies. As we see in literature not many big data sets are
 contain different types of herbs and that's why different
CONV-LA Y:
combinations of crop and weeds create a different research
INPUT RGB
IMAGE 256×3×3
NORMALIZATION
problem to be addressed.

CONV-LA Y:
NORMALIZATION
VI. CONCLUSION & FUTURE WORK
64×3×3
MAX POOL Application of Deep Learning is novel in the field of
NORMALIZATION MAX POOL
autonomous weeding application and promising higher
CONV-LA Y:
1024×3×3 accuracy than any other technique. There exists a potential
CONV-LA Y:
research gap in which deep learning can be applied in various
MAX POOL
512×3×3 crops for identification of weeds and autonomous spray
FC : 4096
application. Spraying herbicide autonomously will benefit
CONV-LA Y:
128×3×3
NORMALIZATION the farmer in getting higher crop yield and control of weeds
FC : 4096 will be more accurate, also soil pollution will be avoided due
to controlled herbicide spray application.
NORMALIZATION MAX POOL
FC : N
REFERENCES
CONV-LA Y:
MAX POOL [1] B. Cheng and E. T. Matson, “A Feature-Based Machine Learning
1024×3×3
Agent for Automatic Rice and Weed Discrimination,” Springer,
Cham, pp.517–527, 2015.
Fig. 4. Hybrid AgroAVNET [16] [2] A. Singh, B. Ganapathysubramanian, A.K. Singh and S. Sarkar,
“Machine learning for high-throughput stress phenotyping in
available for sunflower, carrot, soybean, sugar beet and plants,” Trends Plant Sci. 21 (2), 110–124, 2016.
maize, so acquiring big datasets for these crops and weeds [3] W. S. Lee, V. Alchanatis, C. Yang, M. Hirafuji, D. Moshou, and
C. Li, “Sensing technologies for precision specialty crop
could be a major research contribution. production, "Comput. Electron. Agric.74,2–33, 2010.
The field of application of deep learning in crop and weed [4] T. W. Berge, A.H. Aastveit and H. Fykse, “Evaluation of an
is not saturated i.e. not many such research attempts are algorithm for automatic detection of broad-leaved weeds in
published, so accuracy can be improved, and new deep spring cereals,” Precis. Agric. 9, 391–405, 2008.
[5] E. Hamuda, M. Glavin and E. Jones, “A survey of image
learning models could be obtained which could show higher processing techniques for plant extraction and segmentation in
suitability of application. the field,” Comput. Electron. Agric. 125, 184–199, 2016.
Many crops and weeds combination are not addressed or [6] H. Bakker, Sugar cane cultivation and management, 2012.
little addressed so far in literature, so a wide gap of research [7] A. Ali, J. C. Streibig, S. Christensen and C. Anderson, “Image-
is there to be Researched. No attempt at deep learning based thresholds for weeds in maize fields,” Weed Res. 55(1), 26-
33, 2014.
application is found where the target crops are sugarcane,
rice, wheat, and cotton, which are major crops. Different soils

TABLE IV
Summary of Papers
Deep Targeted Training Setup Training Dataset Dataset Reported
Learning Crop Time Acquisition Setup Strength Accuracy
Type %
Fawakherji Pixel wise Sunflower NVIDIA GTX 1070 Three weeks Nikon D5300 500 images 90
et al. [10] segmentation GPU camera
2019 using CNN
Knoll et al. Image Based Carrot GTX Titan having Not given RGB CAMERA 500 images 93
[11] Convolutional 6GB graphic memory
2018 Neural
Networks
McCool et Image Based Carrot Not mentioned Not given RGB CAMERA 20 training and 90.5
al. [12] Convolutional 40 testing
2017 Neural images
Networks
Tang et al. K-means Soybean Not mentioned Not given Canon EOS 70D 820 RGB 92.89
[13] feature camera images
2017 learning
accompanied
with CNN
Milioto et CNN based Sugar beet NVIDIA GTX1080Ti 200 epochs JAI AD-130 GE 10,000 plant 94.74
al. [14] Semantic in about 48 camera images
2017 Segmentation hours
Andrea et Image Based Maize Core i7 2.7 GHz 8 core Not given Pi camera Version 2835 maize and 92.08
al. [15] Convolutional CPU Computer with 2.1 880 weed
2017 Neural Nvidia GTX950M images
Networks
Chavan et AgroAVNET 12 classes Intel Xeon E5-2695, Not given RGB CAMERA 5544 images 93.64
al. [16] 64GB RAM and
2018 NVIDIA TITAN Xp
with 12GB RAM
[8] P. Qi, X.H. Luo and D.S. Zhang, “Weed recognition based on [13] J. Tang, D. Wang, Z. Zhang, L. He, J. Xin, and Y. Xu, “Weed
digital image processing in wheat field,” J.Xinhua Univ. (Nat. identification based on K-means feature learning combined with
Sci. Ed.), 136-137, 2009. convolutional neural network,” Journal of Computers and
[9] L. L. Wu, J. Y. Liu, Y. X. Wen and X.Y. Deng, “Weed Electronics in Agriculture. Volume (135), 63–70, 2017.
identification method based on SVM in the cornfield,” Trans. [14] A. Milioto, P. Lottes and C. Stachniss, “Real-time blob-wise
Chin. Soc. Agric. Mach. 40(1), 162-166, 2009. sugar beets vs weeds classification for monitoring fields using
convolutional neural networks,” Proceedings of the International
Conference on Unmanned Aerial Vehicles in Geomatics. Bonn,
[10] M. Fawakherji, A. Youssef, D. D. Bloisi, A. Pretto, and D. Nardi, Germany, 2017.
“Crop and Weeds Classification for Precision Agriculture using
context-independent Pixel-Wise Segmentation,” Third IEEE [15] C. C. Andrea, B. B. Mauricio Daniel and J. B. J. Misael, “Precise
International Conference on Robotic Computing (IRC), pp. 146- Weed and Maize Classification through Convolutional Neuronal
152, 2019. Networks,” 2017.
[11] F. J. Knoll, V. Czymmek, S. Poczihoski, T. Holtorf and Stephan [16] T. R. Chavan and A. V. Nandedkar, “AgroAVNET for crops and
Hussman, “Improving e ciency of organic farming by using a weeds classification: A step forward in automatic farming,”
deep learning classification approach,” Journal of Computers and Computers and Electronics in Agriculture. Volume (154), 361–
Electronics in Agriculture. Volume (153), 346–356, 2018. 372, 2018.
[12] C. McCool, T. Perez and B. Upcroft, “Mixtures of lightweight
deep convolutional neural networks: applied to agricultural
robotics,” IEEE Robot. Autom. Lett. 2, 1344–1351, 2017.