Deep Learning for Accurate Real-Time Weed Detection

Subjects: Computer Science, Artificial Intelligence

This article discusses the possibility of accurately detecting the position of weeds in real-time in real conditions. Presented
detailed recommendations for solving the problem with scene density, considered ways for increasing accuracy, and FPS.

Keywords: deep learning in agriculture ; precision agriculture ; weed detection ; robotic weed control ; machine vision for
weed control

1. Introduction
Weeds constitute one of the most devastating constraints for crop production, and efficient weed control is a prerequisite
[1]
for increasing crop yield and food production for a growing world population . However, weed control may negatively
affect the environment [2]. The application of herbicides may result in pollution of the environment because, in most cases,
only a tiny proportion of the applied chemicals hits the targets while most herbicides hit the ground, and a part of them
may drift away [2][3]. Mechanical weed control may result in erosion and harm beneficial organisms such as earthworms in
[4][5]
the soil and spiders on the soil surface . Other weed control methods have other disadvantages and often affect the
environment negatively. Sustainable weed control methods need to be designed only to affect the weed plants and
interfere as little as possible with the surroundings. Weed control could be improved and be more sustainable if weeds
were identified and located in real-time before applying any control methods.

2. Datasets and Image Pre-Processing

A dataset is needed to train neural networks, and image annotation of datasets is one of the main tasks for developing a
computer vision system. Neural networks can be trained via multi-dimensional data and have the potential to model and
extract meaning from a wide range of input information to address complex and challenging image classification and
object detection problems. However, image datasets targeted for training neural networks should contain a significant
number of images and enough variation in the object classes. When tracking an object with an accuracy of a centimeter,
the dataset must be marked up appropriately, which is a highly labor-intensive task in terms of the time and effort required.
At the same time, whichever neural network architecture is used, it is advisable to initialize the neurons using weights
from a trained model that was trained using similar/related objects/images (i.e., the transfer learning technique). Transfer
learning is a way to transfer the understanding (trained weights) of a model trained especially on the large dataset for
initialization on a new dataset with similar objects, or on a smaller training set.

2.2. Image Preprocessing

Image preprocessing (e.g., color space transform, rescaling, flipping, rotation, and translation) is a stage of image
preparation that focuses on creating accurate data in the format needed for the efficient use of image processing
algorithms. The advantages are that these methods do not require high computing power and these processes can be
[6]
implemented in real-time on single-board computers. Faisal et al. presented a good example of a typical preprocessing
process, exploring the use of a support vector machine (SVM) and a Bayesian classifier as an ML algorithm for the
classifying of pigweed (Amaranthus retroflexus L.) in image classification processes. With sufficiently bright color contrast,
this method accurately recognizes the position of the object (Figure 1).
Figure 1. Image processing of pigweed (A. retroflexus): (a) original image, (b) grey-scale image, (c) after sharpening, (d)
with noise filter [6].

Image preprocessing has a broad scope with many different possible filters and different options for transforming an
image. Dyrmann [7] presented a review of various filters for preprocessing but did not give enough examples with images.
[8]
Preprocessing can also be used in real-time. Pajares et al. provided guidance on choosing a vision system for optimal
performance given adverse outdoor conditions with large light variability, uneven terrain, and different plant growth
conditions. Calibration was used for color balance, especially when the illumination changed (Figure 2). This kind of
system can be used in conjunction with DL. Since DL can be trained to show only one illumination, this will reduce the
number of images required to train the model.

Figure 2. Example of auto-calibration process: (a) original image, (b) corrected image [8].

2.3. Available Weed Detection Systems

The real application of machine vision systems is closely related to robotics. Controlling weeds with robotic systems is a
practice that has recently gained increasing interest. In robotic systems, mechanical aspects usually require higher
[9] [10]
precision; hence, detecting weeds from machine vision becomes even more challenging. Qiu et al. and Ren et al.
used popular models of neural networks, such as R-CNN and VGG16, for weed detection with a robotics system. Asha et
al. [11] created a robot for weed control, but realized the challenge of accurate weed detection after deployment in the field
due to the minor differences in size between the weed and the crop (Figure 3). When the weed was close to a cultivated
crop and its leaves (e.g., when they covered each other), the robot had a problem identifying the weeds. After color
segmentation, the weed was perceived as a single object together with the crop.

Figure 3. Examples of artificial images generated by LASSR [12].

[13] [14]
Chang et al. and Shinde et al. used neural networks for image segmentation and the subsequent identification of
weeds by finding the shapes that represent the weed’s structure (Figure 4).

Figure 4. Example of a robot for removing weeds [14].

Dense scenes are the ones where there is dense vegetation, including both crops and weeds, and large occlusion
[15]
between weeds and crops. Zhang et al. presented a review of DL for dense scenes in agriculture. The purpose of this
study is illustrated in Figure 5a, presenting a cluster of fruits on a dense background (including trees from other rows). As
a result, the authors give several general recommendations such as increasing the dataset, generating synthetic data,
and setting hyper-parameters. They concluded that it is advisable to continue increasing the depth of neural networks to
[16]
solve problems with dense scenes. Assad et al. focused on weeds (Figure 5b,c). The authors used semantic
segmentation when conducting DL with a ResNet-50-based SegNet model. As a result, they attained a Frequency-
Weighted Intersection Over Union (FWIOU) value of 0.9869, but the authors did not consider situations where the weed
and the crop came into contact; in such cases, this method will not be efficient.

[15] [16]
Figure 5. Detection of objects within dense scenes: (a) for apples , (b) canola and weeds (initial image) , (c) weeds
[16]
after semantic segmentation .

Proper annotation of the training dataset was crucial to detect the objects precisely. Bounding box-based detection
methods used several IoU (intersection over union) and NMS (non-maximum suppression) values for model training and
inference. Some segmentation methods can be used as well, with blob detection and thresholding techniques for the
identification of individual objects in clusters. Identifying individual objects in clusters is a problem in various fields of
[17] [18]
activity, and much research has been devoted to solving it. Shawky et al. and Zhuoyao et al. considered the
problem of dense scenes and concluded that each situation is unique, and it is advisable to combine various convolutional
neural networks with each other. However, it is very difficult to predict the result in advance because such a method may
[7]
take a lot of time. Dyrmann et al. considered an object as a combination of different objects, depending on their
position, which made the model a little heavier but, at the same time, increased the tracking accuracy. This method is one
of the most promising.

2.2. State-of-the-Art Methods in Weed Detection

[19]
Many scientists have used DL to detect a single feature on a specific background. Rehman et al. considered the role
of CV only in fruits and vegetables among various horticultural products in agricultural areas with statistical machine
[20]
learning technology. Similarly, Osorio et al. considered a DL approach for weed detection in lettuce crops, while
[21]
Ferreira et al. considered weed detection in soybean crops. It is critical that the authors only used a limited dataset
from one field. Therefore, the results of the neural network presented in the work have high accuracy, but at the slightest
change in the detection condition (e.g., a new field or a change of season), the accuracy may change. As a rule, the
results of such works can be used only under consistent conditions, which very rarely happens.

Dokic et al. [22] concluded that DL neural methods were better than classical ML methods for plant classification in images.
The analysis of deep neural networks in the context of agriculture is not fully disclosed because problems with dense
scenes have not been solved. A review by Tian et al. [23] considered practical applications of machine vision in agriculture.
Their analysis showed that computer vision could help in developing agricultural automation for small fields in order to
achieve benefits such as low cost, high efficiency, and accuracy. However, more emphasis was placed on the automation
[24]
process than on machine vision. Khaki et al. proposed the use of a CNN to detect and count corn kernels in images.
The authors trained different models for the detection of objects at high speed under various conditions, lighting, and
angles. The authors applied the standard sliding window approach to detecting the kernel, obtaining high accuracy due to
the correct annotation of the dataset (Figure 6).
 Figure 6. Counting of corn kernels processed in images with CNN [24].

References
1. FAO. NSP-Weeds. Available online: http://www.fao.org/agriculture/crops/thematic-
sitemap/theme/biodiversity/weeds/en/ (accessed on 18 August 2021).

2. Kudsk, P.; Streibig, J.S. Herbicides and two edge-sword. Weed Res. 2003, 43, 90–102.

3. Harrison, J.L. Pesticide Drift and the Pursuit of Environmental Justice; MIT Press: Cambridge, MA, USA; London, UK,
2011; Available online: https://www.jstor.org/stable/j.ctt5hhd79 (accessed on 5 November 2021).

4. Lemtiri, A.; Colinet, G.; Alabi, T.; Cluzeau, D.; Zirbes, L.; Haubruge, E.; Francis, F. Impacts of earthworms on soil
components and dynamics. A review. Biotechnol. Agron. Soc. Environ. 2014, 18, 121–133. Available online:
https://popups.uliege.be/1780-4507/index.php?id=16826&file=1&pid=10881 (accessed on 18 August 2021).

5. Pannacci, E.; Farneselli, M.; Guiducci, M.; Tei, F. Mechanical weed control in onion seed production. Crop. Prot. 2020,
135, 105221.

6. Faisal, F.; Hossain, B.; Emam, H. Performance Analysis of Support Vector Machine and Bayesian Classifier for Crop
and Weed Classification from Digital Images. World Appl. Sci. 2011, 12, 432–440. Available online:
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.390.9311&rep=rep1&type=pdf (accessed on 18 August
2021).

7. Dyrmann, M. Automatic Detection and Classification of Weed Seedlings under Natural Light Conditions. Det Tekniske
Fakultet.University of Southern Denmark. 2017. Available online:
https://pure.au.dk/portal/files/114969776/MadsDyrmannAfhandlingMedOmslag.pdf (accessed on 18 August 2021).

8. Pajares, G.; Garcia-Santillam, I.; Campos, Y.; Montalo, M. Machine-vision systems selection for agricultural vehicles: A
guide. Imaging 2016, 2, 34.

9. Qiu, Q.; Fan, Z.; Meng, Z.; Zhang, Q.; Cong, Y.; Li, B.; Wang, N.; Zhao, C. Extended Ackerman Steering Principle for
the coordinated movement control of a four wheel drive agricultural mobile robot. Comput. Electron. Agric. 2018, 152,
40–50.

10. Ren, G.; Lin, T.; Ying, Y.; Chowdhary, G.; Ting, K.C. Agricultural robotics research applicable to poultry production: A
review. Comput. Electron. Agric. 2020, 169, 105216.

11. Asha, R.; Aman, M.; Pankaj, M.; Singh, A. Robotics-automation and sensor based approaches in weed detection and
control: A review. Intern. J. Chem. Stud. 2020, 8, 542–550.

12. Cap, Q.H.; Tani, H.; Uga, H.; Kagiwada, S.; Lyatomi, H. LASSR: Effective Super-Resolution Method for Plant Disease
Diagnosis. arXiv 2020, arXiv:2010.06499.

13. Chang, C.; Lin, K. Smart Agricultural Machine with a Computer Vision Based Weeding and Variable-Rate Irrigation
Scheme. Robotics 2018, 7, 38.

14. Shinde, A.; Shukla, M. Crop detection by machine vision for weed management. Intern. J. Adv. Eng. Technol. 2014, 7,
818–826. Available online:
https://www.academia.edu/38850273/CROP_DETECTION_BY_MACHINE_VISION_FOR_WEED_MANAGEMENT
(accessed on 18 August 2021).

15. Zhang, Q.; Liu, Y.; Gong, C.; Chen, Y.; Yu, H. Applications of deep learning for dense scenes, analysis in agriculture: A
review. Sensors 2020, 20, 1520.

16. Asad, M.; Bais, A. Weed Detection in Canola Fields Using Maximum Likelihood Classification and Deep Convolutional
Neural Network. Inform. Process. Agric. 2020, 7, 535–545.
17. Shawky, O.; Hagag, A.; Dahshan, E.; Ismail, M. Remote sensing image scene classification using CNN-MLP with data
augmentation. Optik 2020, 221, 165356.

18. Zhuoyao, Z.; Lei, S.; Qiang, H. Improved localization accuracy by LocNet for Faster R-CNN based text detection in
natural scene images. Pattern Recognit. 2019, 96, 106986.

19. Rehman, T.U.; Mahmud, M.S.; Chang, Y.K.; Jin, J.; Shin, J. Current and future applications of statistical machine
learning algorithms for agricultural machine vision systems. Comput. Electron. Agricult. 2019, 156, 585–605.

20. Osorio, K.; Puerto, A.; Pedraza, C.; Jamaica, D.; Rodríguez, L. A Deep Learning Approach for Weed Detection in
Lettuce Crops Using Multispectral Images. AgriEngineering 2020, 2, 32.

21. Ferreira, A.S.; Freitas, D.M.; Gonçalves da Silva, G.; Pistori, H.; Folhes, M.T. Weed detection in soybean crops using
ConvNets. Comput. Electron. Agric. 2017, 143, 314–324.

22. Dokic, K.; Blaskovic, L.; Mandusic, D. From machine learning to deep learning in agriculture—The quantitative review
of trends. IOP Conf. Ser. Earth Environ. Sci. 2020, 614, 012138. Available online:
https://iopscience.iop.org/article/10.1088/1755-1315/614/1/012138 (accessed on 18 August 2021).

23. Tian, H.; Wang, T.; Yadong, Y.; Qiao, X.; Li, Y. Computer vision technology in agricultural automation —A review.
Inform. Process. Agric. 2020, 7, 1–19.

24. Khaki, S.; Pham, H.; Han, Y.; Kuhl, A. Convolutional Neural Networks for Image-Based Corn Kernel Detection and
Counting. arXiv 2020, arXiv:2003.12025v2. Available online: https://arxiv.org/pdf/2003.12025.pdf (accessed on 18
August 2021).

Retrieved from https://encyclopedia.pub/entry/history/show/38796