Field Performance Evaluation Method for a Machine Vision Based Intra Row

Hoeing Machine
Jochen Hemming a,*, Hermen de Jong b, Lauwrens Struik b, Jasper van Meer b, Eldert J. van Henten c
a
Wageningen University & Research, Greenhouse Horticulture, P.O. Box 644, 6700 AP, Wageningen, The Netherlands
b
Machinefabriek Steketee BV, P.O. Box 6, 3243 ZG Stad aan ’t Haringvliet, The Netherlands
c
Wageningen University & Research, Farm Technology Group, P.O. Box 16, 6700 AA, Wageningen, The Netherlands
* Corresponding author. Email: [email protected]

Abstract

Wageningen University and Research developed a machine vision based intra row hoeing machine in
collaboration with Steketee, a manufacturer of agricultural machinery. Although the machine is used in
agricultural practice for several years already there is still a lack of an experimental method to evaluate the field
performance of such a device. Reliable, quantitative information about the performance of the machine in the field
is therefore not known and the effect of variations in field conditions and machine settings on the performance
were not previously researched. The relative number of crop plants damaged by the machine and the relative
amount of soil around the crop plants not hoed were used as criteria to assess the field performance. The
performance was measured both manually on the field and with the help of sensors by logging the actuation
response of the hoeing elements. In the field trial four treatments were combined: 1) the relative number of lettuce
plants, 2) the weed density on the field, 3) the driving speed during hoeing, and 4) the moment of hoeing. Each
possible combination of treatments was repeated three times. It was concluded that the developed method was
suitable to determine statistically significant differences in the hoeing results for the applied treatments. No
difference in the machine performance was found for different numbers of missing crop plants in a plant row. The
machine performance decreased in terms of the relative number of damaged crop plants in case of high weed
density treatments and decreased in terms of the relative number of not hoed intra row spaces in case of high speed
treatments.

1. Introduction

Weeds and crops are competitors in sharing the availability of resources such as light, water and nutrients. The
presence of weeds influences the yield of the crop (Hall et al., 1992; Kropff et al., 1993). The relationship between
weed density and yield loss has been described by many researchers (Cousens, 1985; Spitters et al., 1983; Zimdahl,
2004). Growers want to control weeds, to prevent a decrease of their crop yield. The use of synthetic herbicides
to control weeds in crop production systems is under discussion, due to negative impact on the environment and
human health in case of incorrect use. In North West Europe the use of synthetic herbicides to control weeds is
restricted by legislations in conventional production systems and is not allowed in organic crop production
systems. At the same time the availability of labour in agriculture is decreasing and the costs of labour are
increasing. Therefore, mechanical weed control has become interesting again. Solutions for mechanical weed
control between crop rows are available in the form of all kind of inter row hoeing machines. Moreover, since the
use of electronics and automation became widely accepted in agriculture, the possibility for automated weed
control between single plants in a crop row arose (Hofstee et al., 2014).
Wageningen University and Research developed a machine vision based intra row hoeing machine in
collaboration with Steketee, a Dutch manufacturer of agricultural machinery. The machine, the Steketee IC hoeing
machine, is carried behind a tractor. It’s components and working principles are described in more detail in section
2.1. During the development several prototypes of the machine were tested. In field trials in 2008 it was proven
that it was possible to detect the crop rows and single plant positions for the crops lettuce (Lactuca sativa) and
celeriac (Apium graveolens var. rapaceum). On average less than 1% of the crop plants were not detected and less
than 1% of the weeds were classified as crop. However, the number of weeds was small in the beginning of the
field experiments and more experiments are needed to define the robustness of the system in crops with high and
very high weed intensity (Hemming et al., 2011). During field trials in 2011 it became clear that the due to
differences in field conditions variable distance between the cameras and the crop needs to be taken into account
to achieve a higher hoeing precision. Ultrasonic vertical distance sensors were added to measure the distance to
the crop. Then the machine was tested in sugar beets (Beta vulgaris subsp. vulgaris), endive (Cichorium endivia),
cabbage (Brassica oleracea var. capitata), Brussels sprouts (Brassica oleracea var. gemmifera), celeriac (Apium
graveolens var. rapaceum) and sweet corn (Zea mays). However, there was no quantitative information collected
about the performance in the field (Nieuwenhuizen et al., 2011).
Next to the Steketee IC weeder, more intra row hoeing machines are currently commercially available, e.g.
the Garford Robocrop InRow Weeder (Garford, 2018), the Robovator (Poulsen, 2018) or precision spraying
equipment from Blue River Technology/John Deere (Blue River Technology, 2018). Only very little information

841
is published on evaluation methods for such machines and also about the performance of the machines. One
example of an evaluation of intelligent cultivator for use in lettuce production was described by Smith (2015). In
that research the Robovator and the Steketee IC were compared. Before and after hoeing, the lettuce plants and
weeds were counted. However also in that case, detailed information of the performance per machine was not
described.
Objective
There is still a lack of an experimental method to evaluate the field performance of intra row hoeing machines.
Reliable, quantitative information about the performance in the field is therefore not known and the effect of
variations in field conditions and machine settings on the performance were not previously researched. The
objective of the research described here is to develop and to validate a method to obtain reliable, scientifically
proven, quantitative information about both the performance of the Steketee IC intra row hoeing machine in the
field and the effect of variations in field conditions or machine settings on the performance.

2. Materials and Methods

Intra row hoeing machine

The intra row hoeing machine used in this research is the Steketee IC. It is an intelligent hoeing machine that
is carried behind a tractor, as shown in Figure 181. The machine is equipped with one set of arced shaped hoeing
knives per plant row (Figure 182) that closes in the plant row at the so called intra row spaces to remove unwanted
plants and then opens around the desirable crop plants. The machine contains multiple colour cameras that are
mounted beside each other on the machine, facing straight downwards, under a cover that blocks natural light.
Artificial white LED lighting is used to create a constant level of light under the hood. One of the wheels of the
machine is equipped with an encoder, which generates pulses when the wheel is turning. These pulses are used to
derive the speed and travelled distance of the machine and to trigger the cameras. During hoeing, the colour
cameras take pictures every 0.20 m forwards. The machine uses an ultrasonic sensor to measure the operating
height at the left and right side of the machine and the machine is able to automatically adapt the operating height
at both sides of the machine separately, by changing the height of the support wheels with hydraulic cylinders.
The images acquired by the machine during operation are analysed using computer vision methods. One image
analysis task is to determine the crop row positions. For this task a template fitting algorithm was developed. A
side shift mechanism is used to move the machine to the left or right, to guide the hoeing knives onto the exact
row centres. The second task of the image analysis is to locate every single crop plant in the row. The method
developed for this task is based on a Fast Fourier Transformation (FFT) as described by Bontsema et al. (1991).
More details on the computer vision methods are described in Hemming et al. (2011). The system is designed to
work in all precision sown and transplanted crops, such as for example lettuce, sugar beets and cabbages, and is
able to work in different crop stages.

Figure 181. The Steketee IC hoeing machine mounted behind a tractor (left photo) and visualization of the hoeing knives
(marked by yellow lines) between the crop plants in the row (marked by green dots) (right photo).

842
Figure 182. Hoeing actuator of the Steketee IC hoeing machine. The arc shaped knives can rotate to hoe between single crop
plants in the row. The V shaped hoeing element is for hoeing between the crop rows.

Machine settings that were expected to have a high impact on the machine performance were:

Driving speed: The time needed to pneumatically open or close the hoeing knives is fixed for all circumstances,
assuming a constant air pressure. Thus, the distance that a hoeing knife covers in the driving direction during
opening or closing is depending on the driving speed, even if the algorithm applies a so called speed dependent
correction to the actuating signal. Therefore, both a higher number of damaged crop plants and a higher amount
of not hoed soil around the crop plants were expected for hoeing with a high driving speed compared to hoeing
with a low driving speed.
Virtual safety buffers around crop plant: The machine offers the possibility to set virtual safety buffers around
the detected crop plant. These buffers protect the plants for being damaged by the hoeing knives. The larger the
safety buffer, the earlier a hoeing knife opens before and the later it closes after a detected plant. Thus, a higher
number of damaged crop plants was expected for hoeing with a small safety buffer, compared to hoeing with a
large safety buffer. However, a higher amount of not hoed soil around the crop plants was expected for hoeing
with a high safety buffer, compared to hoeing with a low safety buffer.
Operating height of the machine: As described above the machine is using two ultrasonic vertical distance
sensors to measure and to keep the machine horizontally levelled and to maintain a certain operating height. This
operating height has a big influence on the distance between the cameras and the soil or crop plants. Problems in
the image processing were expected in case of an incorrect height control. Consecutive images taken at a wrong
operating height were expected to result in poorly stitched images, leading to missing or double information. Both
a higher number of damaged crop plants and a higher amount of not hoed soil around the crop plants were expected
for hoeing with a wrong operating height compared to hoeing with a correct operating height.

Experimental field and crop

A field at the experimental farm of Wageningen University and Research was used (GPS coordinates:
51°59'42.4"N 5°39'20.6"E). Lettuce (Lactuca sativa var. capitata) was chosen as crop, since this is the crop in
which the Steketee IC hoeing machine is most used. The experimental field had 16 beds with a width of 1.50 m
and length of 50 m. Each bed had four crop rows besides each other, the inter row distance (the distance between
two crop rows) was 0.375 m. The intra row distance (the distance between two crop plants in one row) was 0.30
m. One bed with a length of 50 m consist of three field plots of ten metre, taking into account a five metre buffer
at the beginning and at the end of the bed and a five metre buffer between each field plot. Each field plot was used
only once, to exclude any influence of a previous weeding action.
In the first half of the experimental field the soil was cultivated already two weeks before planting to allow
weeds to germinate in order to create a high weed density compared to the second half of the experimental field,
were the soil was cultivated just before planting. For both weed densities, half of the field plots were kept fully
planted, the other half of the field plots the crop plants were manually thinned out, 25% of the plants was randomly
removed 12 days after planting (see also Figure 187 and Figure 188).

Field variables that were expected to have a high impact on the machine performance were:

Number of missing crop plants: The core of the used computer vision algorithm is based on a FFT and relies
on a repetitive plant pattern. Missing crop plants will disturb that pattern, leading to a weaker detection of the row
and plant positions. Therefore, both a higher number of damaged crop plants and a higher amount of not hoed soil
around the crop plants were expected for hoeing in fields with a high number of missing crop plants, compared to
fields with a low number of missing crop plants.

843
 Weed density: The algorithm distinguishes crop plants from weeds by the specific contribution of the crop
plants to a certain frequency band, whereas the weeds are expected to contribute randomly to all frequencies. In
case of a high weed density, weed plants could contribute as much to the plant signal as crop plants, causing a
decreased distinction between crop plants and weeds. So, both a higher number of damaged crop plants and a
higher amount of not hoed soil around the crop plants were expected for hoeing in fields with a high weed density,
compared to fields with a low weed density.
Moment in the growing season: A good distinction between crops and weeds was expected early in the growing
season when weeds are small and individual crop plants are not touching or overlapping each other, resulting in a
clear signal. Both the number of damaged crop plants and a higher amount of not hoed soil around the crop plants
were expected for hoeing late in the growing season, compared to hoeing early in the season.

Performance evaluation method

In order to assess the field performance two criteria were used. The first criterion was that an average relative
number of damaged crop plants of at most 1% was considered to be acceptable. The second criterion was that an
average relative amount of soil around the crop plants not hoed by the machine of at most 10% was considered to
be acceptable. Both criteria were based on discussions with several organic growers.

Due to space and time limitations only the following variables were applied to evaluate their effect on the
performance of the machine:
1. Two relative numbers of missing crop plants: 0% and 25% missing crop plants
2. Two weed densities: a normal and a high weed density
3. Two hoeing moments in the growing season: 19 and 23 days after planting
4. Two driving speeds: 0.6 kilometre per hour and 1.8 kilometre per hour
All variations were applied in each possible combination, which means that two relative numbers of missing
crop plants times two weed densities times two moments in the growing season times two driving resulted in 16
treatments. Each treatment was repeated three times which resulted in 48 field plots.

Manual measurements
The weed density was manually counted before and after hoeing using a counting quadrant with a size of one
square metre (Figure 183). Since weed counting is very time consuming, it was decided to count three randomly
selected samples for the normal weed density and three randomly selected samples for the high weed density,
before and after hoeing, both moments in the growing season. The sample locations were randomly selected,
under the condition of a minimal distance of 10 metres between each sample.

Figure 183. Example of weed counting using a counting quadrant before hoeing 19 days after planting

The relative amount of damaged crop plants was manually determined by counting the damaged crop plants.
Counting was done one day after hoeing since at this moment damaged plants were more easily to observe as they
had become dried out or wilted. The relative amount of not hoed soil around the crop plants was manually
determined after hoeing by counting the number of not hoed intra row spaces (the space between crop plants in
one crop row) and the number of not hoed spaces of missing crop plants. The inter row spaces (the space between
two crop rows) were assumed to be always hoed, since the machine has fixed hoeing knives between the different
crop rows. An intra row space was considered as not hoed when all the weeds between two crop plants were
clearly as untouched as before hoeing, an example is shown in Figure 184. A comparison between hoed and not
hoed spaces is shown in Figure 185.

844
 Figure 184. Example of a not hoed intra row space, Figure 185. Comparison between hoed intra row
recognizable by clearly untouched weeds, marked by a spaces, marked by orange circles, and not hoed intra row
yellow circle. spaces, marked by yellow circles.

Sensor-based measurements
The motion of the hoeing knives is actually depending as well on the correctness of the mechanical actuation
of the hoeing knives calculated by the vision algorithm as on the correctness of the response of the hoeing knives.
This means that the performance of the machine was possibly negatively influenced by an inaccurate or incorrect
calculated actuation signal to the hoeing knives or an inaccurate or incorrect response of the hoeing knives, or
both. Therefore, two entities were continuously measured during hoeing and analysed afterwards: the actuation
of the hoeing knives and the response of the hoeing knives.
The actuation signal to the hoeing knives was logged by simply doubling each signal cable to a hoeing knife.
Per hoeing knife, one of the signal cables was plugged into the hoeing knife, whereas the other one was plugged
into an Epec 2024 control unit (Seinäjoki, Finland). The control unit was used to receive the actuation signals
from the cables and send those together with the encoder position every 3 milliseconds to the computer of the
machine via CAN bus. Those actuation signal were logged on the computer. The response of the hoeing knives
was also logged with sensors. Each hoeing knife of the hoeing machine is activated by an pneumatic cylinder. If
the cylinder is retracted, the hoeing knife is open, if the cylinder is extended, the hoeing knife is closed, Vesta
VNPE 3 sensors (Rovigo, Italy) were mounted at the begin and end of each pneumatic cylinder to measure the
presence of the magnetic piston. The values of theses sensors were logged with the same Epec 2024 control unit
described above.
The images taken by the cameras of the machine were also logged, an example image shown in Figure 186.
Both the visualized actuation and the visualised response of the hoeing knives were analysed for the relative
number of damaged crop plants and the relative amount of not hoed soil around the crop plants. Figure 189 in the
result section shows an example of such a visualisation, overlaid on the camera images.

Figure 186. Example of images acquired by the camera system of the machine. Single images are stitched together. The
dashed lines indicate the edges of the single images.

845
 3. Results

Validation of applied field conditions

The relative number of missing crop plants after manually thinning for the field plots with a target of 25%
varied roughly between 20% and 30%. The relative number of missing crop plants for the field plots with a target
of 0% was also slightly different with values roughly between 0% and 5%. This was caused by a few plants that
had died after planting. A Welch’s one-way analysis of variance (ANOVA) was used. This test showed that the
difference between the mean numbers of crop plants at the field plots with 0-5% missing crop plants (96.5 ± 1.6)
and the mean numbers of crop plants at the field plots with 20-30% missing crop plants (74.8 ± 3.0) was
statistically significant, F(1.34) = 972.6 and p = 0.000. Figure 187 shows a plot without missing plants, Figure 188
shows a plot with 25% missing crop plants.

Figure 187. Example of a part of a field plot with no missing crop plants and a normal weed density before hoeing
(left) and after hoeing (right) 19 days after planting.

Figure 188. Example of a part of a field plot with 25% missing crop plants and a normal weed density before hoeing
(left) and after hoeing (right) 23 days after planting.

The overall weed density in the experimental field was very high, an impression is given in Figure 187 and
Figure 188, which is not common on the fields in practice, since growers used to hoe multiple times between the
moment of planting and the moment of harvesting the crop. The weeds that were counted were mostly Orache
(Atriplex) and Redshank (Persicaria maculosa) and to a lower extent Stinging nettle (Utica dioica) and Hedge
bindweed (Convolvulus sepium). The two different weed densities, that were tried to establish by different
moments of soil cultivation as described above were visible by the human eye a view days after planting although,
the difference was ambiguous during the weed counting. However, the weeds growing at the part of the field with
a normal weed density treatment were smaller than the weeds growing at the part of the field with a high weed
density treatment.

846
 Table 79 shows the manually counted number of weeds divided into categories of weeds bigger and smaller
than 5 cm in size, for the two moments of hoeing, for the different weed densities and before and after hoeing.

847