IOP Conference Series: Earth and Environmental Science

PAPER • OPEN ACCESS You may also like

- Changes on chemical composition of
Preliminary study on development of cocoa beans cocoa beans due to combined convection
and infrared radiation on a rotary dryer
fermentation level measurement based on J M Castellanos, C S Quintero and R
Carreno

computer vision and artificial intelligence - Study of Good Handling Practices and
Critical Control Point Determination of
Dried Fermented Cocoa Bean in Gunung
To cite this article: C D Anggraini et al 2021 IOP Conf. Ser.: Earth Environ. Sci. 924 012019 Kidul Regency, Yogyakarta
T F Djaafar, L Elghina, S Widodo et al.

- Identifying the Quality System of Cocoa

Beans to Increase Productivity Using
Backpropagation Neural Network
View the article online for updates and enhancements. Algorithm: A Case Study at Sumber Rejeki
Farmers Group, Patuk Gunung Kidul
U Lestari, R A Kumalasanti and E S
Wulandari

This content was downloaded from IP address 181.214.12.99 on 09/12/2021 at 01:22

2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

Preliminary study on development of cocoa beans

fermentation level measurement based on computer vision
and artificial intelligence

C D Anggraini1, A W Putranto1,2, Z Iqbal2, H Firmanto3 and D F Al Riza 1,2*

1
Bioprocess Engineering Study Program, Faculty of Agricultural Technology,
Universitas Brawijaya, Jl. Veteran, Malang, Indonesia
2
Agricultural Engineering Department, Faculty of Agricultural Technology,
Universitas Brawijaya, Jl. Veteran, Malang, Indonesia
3
Indonesian Coffee and Cocoa Research Institute (ICCRI) Jl. PB. Sudirman No. 90,
Jember, Indonesia

E-mail: [email protected]

Abstract. The fermentation process is an important indicator of cocoa beans' quality. The
standard method used is the Magra test by splitting the cocoa beans and observing the color of
the beans with the naked eye to estimate the degree of fermentation. Although, manual estimation
systems require specific expertise, which can lead to inconsistency in predicting cocoa bean
fermentation rate. This research aims to develop a classification model of two categories of
cocoa, i.e., fermented and unfermented cocoa, using computer vision and a machine learning
model. Image analysis has been carried out, and color features have been used to train and
compare several classification models. After analyzing the data, it was found out that a model
that can quantify the standard and accurate measurement of the degree of fermentation of cocoa
beans using artificial neural network models so that it can segment, calculate, and grade
classification by using color feature extraction, which is the average value of RGB and L*a*b.
The Artificial Neural Network (ANN) Multilayer Perceptron (MLP) has been found to be
superior compared to other models achieving training and validation accuracy of 94%.

1. Introduction
Cocoa is one of the plantation crops that is widely cultivated in tropical areas such as Indonesia. The
high cocoa production capacity makes cocoa a national leading commodity that contributes the third
largest foreign exchange contribution of 1.2 billion USD. The prospect of the world cocoa market is
experiencing an increase in market demand throughout the world, especially in Asia and Africa, by 38%
and 72%, respectively [1]. The quality of cocoa beans is highly dependent on several factors, especially
on post-harvest processing. The post-harvest processing of cocoa includes several processes, especially
the fermentation process, which greatly affects the quality of the final product, such as aroma precursors
and the cocoa flavor[2].
It is unfortunate that the high production of cocoa beans from Indonesia is not balanced with the low
quality of cocoa beans. This is due to the lack of knowledge of farmers in post-harvest processing,
especially in the uncontrolled fermentation process. Generally, cocoa beans of good quality must be
Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution
of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Published under licence by IOP Publishing Ltd 1
2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

perfectly fermented with a brownish-red color and have a moisture content of about 7.5% and have gone
through a process of sorting the content of damaged beans. Fermented cocoa beans will experience a
color change due to a decrease in the content of polyphenols (including anthocyanidins) which are
oxidized and polymerized into insoluble high molecular weight compounds (tannins). In addition,
fermented cocoa beans showed a decrease in pH and inactivation of enzymes that produced flavor
precursors [3]. According to Davit M et al.[4], 85% of nationally produced cocoa beans are not
fermented so that it has an impact on the selling price of cocoa beans in the international market. The
application of the fermentation process is an important indicator to determine the quality and price of
cocoa beans from Indonesia in order to compete in the world market [5]. Therefore, it is necessary to
improve the process, both for the fermentation process and for measuring the quality of fermentation
that is effective and efficient and needs to be widely available in Indonesia.
The application of manual and uncontrolled fermentation technology among farmers using wooden
boxes poses a big challenge, especially in the process of measuring the degree of cocoa fermentation.
This is because the environmental conditions of each wooden box are different, and the physical changes
of each sample [6]. Fast and accurate measurement methods are needed in measuring the degree of
fermentation. The standard method for evaluating the degree of fermentation of cocoa beans is to use a
cutting test in the form of a bean cutting device called a Magra, and then an expert observer will assess
the degree of fermentation of cocoa beans visually by observing the color of the flesh on beans that have
been split open. This observation requires experts, and subjectivity can occur because the measurement
of the level of fermentation is carried out qualitatively [7]. Along with technological developments,
methods for measuring cocoa quality have emerged using spectrophotometry, chromatography, Fourier
transforms infrared spectroscopy, and imaging systems. However, this method has several obstacles if
applied on a large scale, such as it is time-consuming, requires a lot of human resources, expensive
equipment prices, and is difficult to apply to practitioners, especially cocoa farmers [2].
Computer vision is a technology that is currently being developed as an effort to measure the quality
of agricultural products, which has the advantage of being cheaper, faster, accurate, and non-destructive
[2]. The use of the image analysis method with multivariate statistics has been successfully applied to
agricultural products such as fruits and vegetables. In addition, the use of computer vision has a high
accuracy rate of > 90% by using several methods such as Support Vector Machines (SVM), C4.5,
AdaBoost (AB), k-Nearest Neighbors (KNN), Logistic Regression (LR), Stochastic Gradient Boosting
Trees (GBDT), Extreme Learning Machines (ELM), Sparse Representation-based Classification (SRC),
and Deep Learning (DL). The use of computer vision aims to produce a fast and accurate measurement
process for classifying and predicting the quality of an agricultural product [3]. Based on this problem,
this article aims to develop an image processing, feature extraction procedure, and a classification model
of two classes of the cocoa bean, i.e., fermented and unfermented cocoa using computer vision and
several machine learning models. This model is used as the first step in making a portable fermentation
index measurement system so that it can be developed into a system that can be widely applied in the
community.

2. Material and methods

The research was conducted from May to June 2021, using a mini studio located in the Faculty of
Agricultural Technology, University of Brawijaya. The sample of cocoa beans (Theobroma cacao L.
cv. Trinitario) was obtained from the Indonesian Coffee and Cocoa Research Institute, Jember,
Indonesia. The samples of cocoa beans analyzed were 750 samples of the fermented and unfermented
beans category (Figure 1). The fermented samples have been fermented for 4-5 days at 55 ºC and dried
for 1-3 days to obtain a moisture content of 7.5%. Note that the surface appearance could not be used to
determine the fermentation level. Thus, the samples were cut, and the color of the internal beans was
observed.

2
2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

Figure 1. Cocoa beans sample (before cutting).

2.1. Image acquisition

The image acquisition process is carried out by taking image data using standard lighting (mini studio)
and a color chart as a reference. The type of camera used includes a DSLR camera using ISO 100, an
aperture value of F5.6, and a manual exposure value of 1/13. Primary data used in the form of image
data of cocoa beans from two categories, namely fermented and unfermented, as many as 140 images,
and each image has 5 pieces of split cocoa beans. The image used is an image file with a format Joint
Photographic Group (.JPG) measuring 3456 x 5184 pixels with a depth of 72 dpi, as shown in Figure 2.
The mini-studio and the result of image acquisition used to look like the picture below.

Figure 2. Cocoa beans sample (after the cut).

2.2. Image processing and color feature extraction

Pre-processing data aims to change the image size by 30% from 5184 x 3456 pixels to 1555, 1036. In
addition, the pre-processing of this data includes the process of converting BGR (Blue, Green, Red)
colors to RGB (Red, Green, Blue). Color parameters used in feature extraction include RGB and L*a*b
colors. Pre-processing can be carried out using standardized binarization in the separation of different
cocoa bean backgrounds [8]. The pre-processing stage is carried out by changing the size of the RGB
image by 40% to facilitate the process of converting RGB color to HSV[9]. This stage aims to convert
the RGB image to HSV (Hue, Saturation, Value). Hue images show colors such as red, blue or yellow,
while saturation is used to determine how pure the color of the image is, and the value shows a measure
of how much brightness/light comes from color. The image results after converting to black and white
into one threshold value will cause missing information. The Otsu method is useful for automatically

3
2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

dividing the histogram image into two different areas. Using a discriminant analysis approach by
determining a variable that can distinguish between two or more groups that appear. The formulation
used is as follows: for example, the threshold value to be searched is with the letter k. The value of k
ranges from 1 to 255 [9]. The Otsu method is a segmentation method that uses a histogram to group the
pixels in the x- and y-axis images. The x-axis in the Otsu method is a different level of intensity, while
the y-axis represents the number of pixels that have that intensity value [8].
Feature extraction is the process of taking the color you want to analyze, including RGB and Lab
colors. The value of each sample of cocoa beans Characteristics from the images was carried out because
it was estimated that they would have different values for each class of cocoa beans fermentation level.
Taking these values is carried out on all image pixels that are included in the segmentation so that only
the value of the fruit image is obtained without the image background [10]. CFs include color mean
value and excess RGB index. Color mean value can be described as follows [11]:
1
Color mean value = 𝑀 ∑𝑀
𝑖=1 𝐶𝑜𝑙𝑜𝑟 𝑣𝑎𝑙𝑢𝑒
(1)

where: color value can be defined as the range of each color space in the pixel such as red, green, blue,
grey, L*, a*, b*. M is the total number of pixels in the masked region of the image. The complete image
analysis step has been shown in Figure 3.

2.3. Data analysis

Image and data processing, training, and validation of the machine learning model have been carried out
using Python 3.8 in Jupyter Notebook, with Tensorflow and Keras libraries. Modeling This model is
formed from analyzing 6 color features from 1443 datasets which are divided into 2 categories of cocoa
beans. The dataset is divided into train data and test data with a ratio of 8.5:1.5. Several machine-learning
models, including Logistic Regression, Decision Tree, Random Forest, Support Vector Machine (SVM),
and Artificial Neural Network Multilayer Perceptron (ANN-MLP), have been trained, and the results
have been compared. The hyperparameter has been optimized for each model. For the ANN-MLPP
models 2, 3 and 4, hidden layers have been compared with a different number of nodes. A learning rate
of 0.001 has been used, and other parameters, i.e., batch and epoch number, have been tweaked. The
activation function of ReLU has been used for the hidden layer, and Sigmoid has been used for the
output layer. The weight vector value initialization method uses random values and Adam optimizer has
been used to optimize the model [12]. Binary cross-entropy and binary accuracy have been monitored
through the training and validation process.

Figure 3. Image processing step.

4
2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

3. Results and discussion

3.1. Image pre-processing and features extraction

Based on the results of the analysis of 1443 images of cocoa beans from both of category, it shows that
there are differences in the value of each color feature. After preprocessing feature extraction, the color
feature value of each sample has been obtained as shown in Table 1.

Table 1. Dataset with a color feature value.

Picture R_ave G_ave B_ave L_ave a_ave b_ave Fermented
0 35.29186 25.01141 20.6848 25.32251 85.69086 77.12148 1
1 37.37944 25.08545 20.10656 25.46735 86.78497 76.37036 1
2 28.69425 22.48285 19.14682 21.71258 89.91746 84.99385 1
…… …… …… …… …… …… …… ……
1441 25.08676 11.62848 5.961068 10.16908 114.7531 102.3137 0
1442 26.40409 13.20462 9.59924 13.30772 85.45105 73.47752 0
1443 20.04759 11.6034 7.57209 11.38585 78.64546 71.67397 0

The threshold analysis will provide a threshold value for each image between the foreground and the
background. The threshold value uses half the value of the global threshold automatically to reduce
noise from each image category. In addition, in doing image segmentation, it is known that several
different results are known when analyzing images directly from RGB colors and when doing the
grayscale conversion process from each category. From Otsu's method, this determines the threshold
value based on stats image information of cocoa beans from each category by minimizing the number
of weighted group variances, where the weight is the probability of each group[13]. According to Bhahri
and Rachmat [14] in the segmentation process, the first step is to divide the image into several parts to
determine the boundaries and continue with the stage of giving an index to the color of each pixel that
shows the part in a segmentation. Otsu's method works by separating objects with a discriminant analysis
background that will maximize the variables visualized in a histogram. The histogram is used to
determine the number of pixels and the threshold for each gray level.

3.2. Machine learning models

The average value of each color channel of the masked area of the images has been labeled as R_ave;
B_ave; G_ave, L_ave, a_ave, and b_ave. Based on research conducted by Oliveira, et al., [3] explained
that the application of RGB imagery in classifying the fermentation rate of cocoa beans has several
advantages such economical and can thus be applied by farmers and small processors as a low-cost
alternative to the traditional cut-off test with accurate prediction results on the fermentation grade of
cocoa beans. These features then were used as an input of a machine learning model. For the machine
learning model other than ANN, we tried to compare the result of the model with data scaling (standard
scaler) and with no scaling. Table 2 shows the training and validation accuracy results of several
machine learning models besides ANN. Among those models, none could achieve more than 90%
accuracy.

3.3. Prediction using an artificial neural network model

ANN-MLP model with different architecture has been trained and validated. From these data the value
in making the classification and modeling of an artificial neural network is influenced by several factors
such as the selection of the number of neurons in the hidden layer and the appropriate learning rate value
of each data classification model. The addition of the number of neurons in the hidden layer, the longer
the running time of data processing, but will increase the accuracy value obtained. Addition and
subtraction of a matrix can be obtained using the confusion matrix which will affect the accuracy and

5
2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

error values obtained [15]. The method which is simple but with high accuracy will be preferable to
avoid model complexity curse. The simple model also mean that it could be executed faster with lower
memory and storage required.
Table 2. Machine learning model.
Training Validation
No. Model Scaling
Accuracy Accuracy
Logistic Regression (with L2
1 None 70.13 76.47
regularization)
Logistic Regression (with L2
2 Standard Scaler 60.52 64.71
regularization)
3 Decision Tree (criterion: entropy) None 100 85.81
4 Decision Tree (criterion: entropy) Standard Scaler 100 85.81
5 Random Forest (n_estimators: 30) None 99.74 86.85
6 Random Forest (n_estimators: 30) Standard Scaler 99.83 85.12
7 SVM (kernel: RBF, gamma=1.7) None 100 59.86
8 SVM (kernel: RBF, gamma=1.7) Standard Scaler 89.35 84.78

From the results, it has been found that the first model which is using 2 hidden layers (16-16) provide
the best results (Figure 4). Interestingly ANN-MLP model which used more than two layers, i.e., three
and four could not achieve accuracy more than the two hidden layers models. From the results of the
preparation of the model using artificial neural network, there are several important points. Factors to
choose the type of activation function, the number of iterations and the number of hidden layers of each
model need to be considered. In addition, the number of hidden layers must also be adjusted to the
learning rate of a model. ANN modeling using these 3 models is very capable of being developed in a
system to classify the level of cocoa bean fermentation with an accuracy value of >90%. Table 3 shows
the results of the accuracy and validation of the best model for each ANN MLP architecture.
Table 3. The results of artificial neural network models (data training=85%, data validation=15%).
No. ANN Hyperparameter Scaling Training Validation Training Validation
Accuracy Accuracy Loss Loss
1 Hidden layer: 16-16 Standard 0.942 0.94 0.145 0.176
Activation: relu-relu-sigmoid Scaler
Learning rate: 0.002
Epoch: 500
Batch size: 15
2 Hidden layer: 16-16-8 Standard 0.936 0.931 0.152 0.184
Activation: relu-relu-sigmoid Scaler
Learning rate: 0.001
Epoch: 500
Batch size: 15
3 Hidden layer: 16-16-8-8 Standard 0.942 0.921 0.127 0.198
Activation: relu-relu-sigmoid Scaler
Learning rate: 0.001
Epoch: 500
Batch size: 15

6
2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

Figure 4. Model accuracy and loss during training and validation for ANN model with 2 hidden layers
and nodes 16-16.

4. Conclusions
From 1143 cocoa beans image data from two categories of fermented and unfermented cocoa beans, it
is obtained if there is a color relationship, especially RGB in determining the correlation value between
parameters. The use of Otsu's method will simplify the image segmentation process for each category
due to the loss of noise in the form of light disturbances and unwanted image objects to be detected.
Image segmentation is the first step in developing a detection system for the degree of fermentation of
cocoa beans through image images by dividing an image into several regions, which in an area have
similar attributes. In making modeling using ANN, it is necessary to pay attention to several factors such
as the number of iterations, the value of learning rate and the number of neurons in a hidden layer. From
the model that has been made to classify the level of fermentation of cocoa beans using an artificial
neural network, it has an accuracy value of >90% from 2 categories of fermentation levels. Especially
in the 1st model with 2 hidden layers at node values (16,16) and 0.002 learning rate value, through the
output activation function in the form of Sigmoid, has the highest validation accuracy value of 94%.
These results show that it is relatively easy to classify the fermented and unfermented cacao beans with
any machine learning model using a few color features. Further challenges are to predict the
fermentation levels in more detail in higher resolution using classification with more classes or a
regression model.

References
[1] Maulana A and Kartiasih F 2017 Analisis ekspor kakao olahan indonesia ke sembilan negara
tujuan tahun 2000 – 2014 (Analysis of indonesian cocoa exports to nine destination countries
2000 – 2014) J. Ekon. Pembang. Indones. 17 2 103-17
[2] Yro A, Edié C N, Kpalma K, Aubain Y, Edié Camille Z and Kidiyo K 2018 Cocoa beans
fermentation degree assessment for quality control using machine vision and multiclass svm
classifier Int. J. In-Novation Appl. Stud. Issr J. 24 4 1711-7
[3] Oliveira M M, Cerqueira B V, Barbon S and Barbin D F 2021 Classification of fermented cocoa
beans (cut test) using computer vision J. Food Compos. Anal. 97 103771
[4] Davit J, Yusuf R P and Yudari D A 2013 Pengaruh cara pengolahan kakao fermentasi dan non
fermentasi terhadap kualitas, harga jual produk pada unit usaha produktif Tunjung Sari,
Kabupaten Tabanan (The influence of fermented and non-fermented cocoa processing
methods on the quality, selling price of products in the productive business unit of Tunjung
Sari, Tabanan Regency) E-Journal Agribisnis dan Agrowisata 44956
[5] Nurhayati N, Utami R R and Yusdianto Y 2019 Teknologi digital sensor warna untuk mengukur
tingkat fermentasi kakao (Ulasan) (Digital color sensor technology to measure cocoa
fermentation rate) J. Ind. Has. Perkeb. 14 2 16-23

7
2021 International Conference on Green Agro-industry and Bioeconomy IOP Publishing
IOP Conf. Series: Earth and Environmental Science 924 (2021) 012019 doi:10.1088/1755-1315/924/1/012019

[6] Yulianti N L and Arda G 2018 Studi kombinasi lama fermentasi, jenis wadah dan suhu
pengeringan terhadap karakteristik kakao kering (Study of combination of fermentation time,
type of container and drying temperature on the characteristics of dry cocoa) J. Ilm. Teknol.
Pertan. Agrotechno 3 1 304-8
[7] Tan J, Balasubramanian B, Sukha D, Ramkissoon S and Umaharan P 2019 Sensing fermentation
degree of cocoa (Theobroma Cocoa L.) beans by machine learning classification models based
electronic nose system J. Food Process Eng. 42 6 e13175
[8] Putra E D and Santosa S 2017 Optimasi kemampuan segmentasi otsu pada identifikasi plat nomor
kendaraan indonesia menggunakan metode gaussian (Optimization of otsu's segmentation
capability on identification of indonesian vehicle number plates using the gaussian method)
Pseudocode 4 1 47-60
[9] Muwakhid I A and Nurdiyah D 2018 Otsu method for image finish segmentation with
components of hue saturation value 15 67-73
[10] Prahudaya T Y and Harjoko A 2017 Metode klasifikasi mutu jambu biji menggunakan knn
berdasarkan fitur warna dan tekstur (Guava quality classification method uses knn based on
color and texture features) J. Teknosains 6 2 113-23
[11] Hendrawan Y and Al Riza D F 2016 Machine vision optimization using nature-inspired
algorithms to model sunagoke moss water status Int. J. Adv. Sci. Eng. Inf. Technol. 6 1 45-57
[12] Agmalaro M A, Kustiyo A and Akbar A R 2013 Identifikasi tanaman buah tropika berdasarkan
tekstur permukaan daun menggunakan jaringan syaraf tiruan (Identification of tropical fruit
plants based on leaf surface texture using artificial neural networks) J. Ilmu Komput. Agri-
Informatika 2 2 73-82
[13] Yousefi J 2015 Image binarization using otsu thresholding algorithm Res. Gate, DOI:
10.13140/RG.2.1.4758.9284
[14] Bhahri S 2018 Transformasi citra biner menggunakan metode thresholding dan otsu thresholding
(Binary image transformation using thresholding and Otsu thresholding method) E-Jurnal
Jusiti (Jurnal Sist. Inf. Dan Teknol. Informasi) Doi: 10.36774/Jusiti.V7i2.254.
[15] Rahman M F, Alamsah D, Darmawidjadja M I and Nurma I 2017 Klasifikasi untuk diagnosa
diabetes menggunakan metode bayesian regularization neural network (Rbnn) (Classification
for diagnosing diabetes using the Bayesian regularization neural network method) J. Inform.
11 1 36