Received 11 August 2024, accepted 28 August 2024, date of publication 30 August 2024, date of current version 17 September 2024.

Digital Object Identifier 10.1109/ACCESS.2024.3452552

Coffee Bean Defects Automatic Classification

Realtime Application Adopting Deep Learning
HONG-DANH THAI 1,2 , HAN-JONG KO3 , AND JUN-HO HUH 2,4 , (Member, IEEE)
1 Department of Data Informatics, National Korea Maritime and Ocean University, Busan 49112, Republic of Korea
2 Department of Interdisciplinary Major of Ocean Renewable Energy Engineering, National Korea Maritime and Ocean University, Busan 49112, Republic of

Korea
3 Department of Agricultural Sciences, Korea National Open University, Seoul 03087, Republic of Korea
4 Department of Data Science, National Korea Maritime and Ocean University, Busan 49112, Republic of Korea

Corresponding authors: Han-Jong Ko ([email protected]) and Jun-Ho Huh ([email protected])

ABSTRACT The coffee industry contributes to the economic restructuring of many countries, often
associated with a closed process from production to consumption. The green coffee bean grading standard
provided by the Specialty Coffee Association (SCA) is one of the best methods for grading coffee beans.
Traditionally, the assessment of quality and classification of coffee beans relies on visual examination, which
demands significant time and effort and is easily inaccurate. Deep learning technology, characterized by
precision, velocity, and veracity, can be adopted to empower the reduction of human labor and improve
the productivity, quality, and efficiency of these tasks. Therefore, this paper aims to address these issues by
implementing deep learning to classify coffee bean quality in real time by integrating the system with a cloud-
based solution. First, image processing and data augmentation techniques are employed to handle the coffee
bean image data. Subsequently, the model is trained using YOLOv8, a framework for object recognition,
and OpenCV, an open-source image processing technology, to classify coffee beans. Finally, an application
is developed for real-time video and image-streaming coffee bean recognition using React Native, NodeJS,
and Python. The experimental results provide empirical evidence that our system enhances accuracy and
efficiency in the tasks of classifying coffee bean quality in nine distinct varieties of coffee beans, with the
time required reduced to a mere 1 to 3 seconds. Our system can be a useful solution for coffee producers,
processors, and traders without relying on stationary equipment, especially in large farms or warehouses.

INDEX TERMS Coffee bean defects, quality classification, computer vision, YOLOv8, OpenCV, cloud-
based application, deep learning.

I. INTRODUCTION industry, including structural supply changes in quantity and

Coffee is regarded as one of the most extensively traded quality, require farmers to ensure high-quality coffee pro-
commodities globally and has become a primary export duction to stay competitive in this evolving market. Coffee
of economic significance for developing countries such as beans must be carefully selected before packaging and mass
Brazil, Vietnam, Colombia, Indonesia, and Ethiopia [1]. The production. Grading green coffee beans is an important stage
coffee industry contributes to the financial restructuring of in coffee import or export [4]. The green coffee bean grad-
many countries, often associated with a closed process from ing standard provided by the Specialty Coffee Association
production to consumption [2]. Approximately 20-25 million (SCA) is one of the best methods for grading coffee beans.
small-scale agriculturalists in over 50 developing countries Traditionally, the assessment of quality and classification of
cultivate and commercialize coffee, which accounts for 20% coffee beans relies on visual examination, which demands
of their export revenues [3]. Market changes in the coffee significant time and effort and is prone to manual inaccu-
racies. This leads to labor costs, and quality control cannot
The associate editor coordinating the review of this manuscript and be guaranteed due to various objective factors, such as the
approving it for publication was Yiqi Liu . inability to identify small samples through the human eye

2024 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
VOLUME 12, 2024 For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ 126503
 H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

and the impact of poor physical conditions on small sample • Implementation of an experimental prototype as a web
control. There is a need for more objective, automated, and application and a mobile application for automatic clas-
efficient methods to assess coffee bean quality and classify sification in real time.
them accurately. The application of modern technologies can The rest of the paper is organized in the following man-
help reduce labor costs and minimize the impact of objective ner. Section II provides an overview of existing research on
factors in these tasks. Deep learning technology, character- detecting and classifying coffee beans. Section III provides a
ized by precision, velocity, and veracity, can be adopted. For proposed research design and experiment setup for an auto-
example, computer vision, in which many different machine matic classification real-time application utilizing Yolov8.
learning and deep learning algorithms, such as convolutional The implementation outcomes of the automatic application
neural networks (CNNs) [5], [6], [7], [8], support vector are presented in Section IV. In Section V, an analysis of
machines (SVMs) [9], [10], [11], or K-Nearest Neighbor performance and evaluation will be shown compared to alter-
(K-NN) [12], [13], are trained on labeled data to classify native systems. The discussion of the conclusions and future
coffee beans into different grades or quality categories. The work, to enhance the results, can be found in Section VI.
advantages of these technologies over traditional methods lie
in their ability to be easily replicated across multiple factories. II. RELATED WORKS
Moreover, Artificial Intelligence (AI) technology can detect A. COFFEE BEAN QUALITY CLASSIFICATION
and self-learn from anomalous factors, leading to continuous The coffee industry faces fluctuations, requiring farmers to
improvement in the detection process. produce high-quality coffee to stay competitive. Coffee clas-
However, there are certain limitations to consider. While sification is a way of assessing the quality of green coffee
earlier algorithms could identify simple faults such as broken beans before they are exported to produce uniform com-
or black beans, detecting subtler or more complex defects was mercial shipments that meet specific quality criteria, thereby
more challenging. To accurately detect and classify defects creating the basis for a reasonable pricing system [15].
like mold or insect damage, additional analysis or specialized However, there is no standard worldwide classification and
imaging techniques may be required. In addition, imple- evaluation system for coffee. Different producing countries
menting computer vision systems for coffee bean grading have their systems. The most widely recognized coffee classi-
can involve significant upfront costs, including hardware, fication system was developed by the Specialty Coffee Asso-
software development, and integration with existing man- ciation (SCA), formerly the Specialty Coffee Association of
ufacturing lines. Financial barriers may hinder small-scale America (SCAA) [16]. The SCA system uses several key
coffee farmers and processors from adopting these criteria to evaluate and grade green coffee beans, including
technologies [14]. defects, screen size, moisture content, cup quality, and color.
Therefore, this paper aims to address these issues by devel- In this research, we mainly focus on coffee bean defects
oping a coffee bean automatic classification real-time mobile classification, which can be recognized on the surface of
application adopting deep learning technology for SMEs and the coffee beans. There have been reports of more than
agriculturalists. First, integrating deep learning-based classi- 13 categories of defects used to classify coffee beans [17],
fication into our mobile application allows for on-site, instant [18]. However, in this research, we only focus on frequent
assessment of coffee bean quality. Our mobility and acces- 9 specific types of defects to evaluate the quality of Robusta
sibility of the proposed system can improve effectively and coffee beans. We will explain the bean defects and their
replace traditional desktop-based or specialized hardware. corresponding characteristics below.
Second, our mobile application easily serves a wider range 1. Black - Full Black is considered a primary defect in
of users, including smallholder farmers, cooperatives, and coffee when a bean is predominantly black in color. This
quality control personnel, SMEs. This enhances the reach defect occurs due to the over-fermentation of pigments [19].
and adoption of the technology, particularly in remote or 2. Broken - Broken beans commonly occur when the pulp-
resource-constrained coffee-producing regions. Thirdly, our ing or dry milling process is flawed, typically due to incorrect
mobile-based solution can help to reduce operation costs equipment adjustments or excessive friction and pressure
compared to traditional specialized hardware or desktop- applied to the beans [19].
based systems. To sum up, we want to minimize the research 3. Big Broken - The primary defect known as ‘‘Big
gap by developing a highly precise and user-friendly mobile Broken’’ occurs when a significant portion of the beans
application for coffee bean automatic classification in real- are predominantly large broken pieces. This defect typically
time. This application has the potential to influence the arises during the pulping or dry milling processes.
coffee industry’s quality control, traceability, and overall 4. Immature - The immature or unripe bean fails to fully
effectiveness. develop due to various factors, such as improper picking of
The main contributions of the paper can be demonstrated unripe cherries and uneven ripening in late-ripening varieties
as follows: cultivated at high altitudes [19].
• Developing a coffee bean quality classification based 5. Insect - Insect defects can occur during cultivation,
on deep learning technology, which can identify and harvesting, processing, transportation, and storage, resulting
distinguish nine distinct varieties of coffee beans. in physical damage, such as holes or molds.
126504 VOLUME 12, 2024
H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

6. Mixed - Mixed defects occur when a single bean or a the right image processing and classification method can
group of beans exhibit multiple defects simultaneously. These ensure greater accuracy in the coffee quality assessment
defects can include broken beans, black beans, insect damage, process.
mold, or other imperfections. Hidayat et al. [21] explore the coffee roasting process
7. Mold - Mold defects refer to the growth of mold or and levels by image processing. It employs the Local
fungal organisms on coffee beans during various stages of Binary Pattern (LBP) method to extract structural features
coffee production, such as harvesting, processing, or storage. from images and a Naive Bayes classifier for classification.
8. Brown - Brown is considered a primary defect in coffee LBP is a simple and effective method, while Naive Bayes
when a bean is predominantly black in color. This defect is a probability-based classification method that assumes
occurs due to the over-fermentation of pigments. attributes are independent. Test results show an 81% accuracy
9. Partly Black - Partly Black is classified as a secondary rate, indicating a promising performance for image-based
defect in coffee, characterized by approximately half of a classification systems.
bean being covered in black discoloration. This defect occurs The paper [22] classifies ripe and unripe coffee beans
due to the over-fermentation of pigments, which is linked to using structural features and the K-Nearest Neighbor (KNN)
the presence of microorganisms [19]. algorithm. The research team used image processing tech-
Historically, the evaluation of coffee bean quality and niques to extract structural features from images of coffee
categorization has been based on visual inspection, which beans and then employed MATLAB’s Classification Learner
requires substantial time and effort and is susceptible to App for classification. The results demonstrate that KNN
human errors. As a result, labor expenses are incurred, and exhibits the highest performance, achieving 97% accuracy
ensuring quality control becomes challenging due to objec- and fast processing speed. Textural features outperform color
tive factors, including the human eye’s incapacity to detect features in distinguishing between ripe and unripe coffee
small samples and the negative effects of unfavorable phys- beans.
ical circumstances on small sample management. There is Research [23] presents the classification of defects in
a requirement for further objective, automated, and efficient Robusta coffee beans using the YOLO algorithm. This
techniques to evaluate the quality of coffee beans and appro- research aims to build a system to automatically identify
priately categorize them. defects in coffee beans by using image processing to identify
broken, black, and normal coffee beans. The system is imple-
B. DEEP LEARNING BASED ASSESSMENT TECHNOLOGY mented using the You Only Look Once (YOLO) algorithm
The application of modern technologies can help reduce labor and the latest version of the algorithm (YOLOv5). The total
costs and minimize the impact of objective factors in the accuracy of the system is 95.11%. This system will assist
classification of coffee beans. Computer vision, in which coffee growers and consumers, as they can use the coffee bean
many different machine learning/ deep learning algorithms defect identification system to detect defects.
were applied to the classification of coffee beans in pre- In the paper [24], the authors focus on developing a real-
vious research. Deep learning technology, characterized by time error-checking algorithm for green coffee using NIR
precision, velocity, and veracity, can be adopted. The advan- hyperspectral imaging. The paper addresses the issue of
tages of these technologies over traditional methods lie in checking coffee beans for defects, including defects such as
their ability to be easily replicated across multiple factories. black beans, mold, insect damage, breakage or damage. The
Moreover, artificial intelligence (AI) technology can detect research paper combines near-infrared hyperspectral sens-
and self-learn from anomalous factors, leading to continuous ing and deep learning to create a real-time error-checking
improvement in the detection process. Ongoing research and algorithm for sorting defective green coffee beans. Three con-
advancements in machine learning algorithms, imaging tech- volutional neural networks (CNN) were designed to achieve
nologies, and data analytics contribute to improving accu- real-time inspection. Experimental results show that the over-
racy and efficiency in coffee bean grading systems. These all accuracy reaches 98.6% on 1026 green coffee samples.
advancements play a crucial role in producing high-quality The algorithm also achieved a Kappa value of 97.2% and
coffee products and fostering the growth of the coffee real-time sorting speed.
industry. The paper [25] introduces a model using Deep Learning
The paper [20] emphasized comparing the performance methods, especially Convolutional Neural Networks (CNN)
of classification algorithms to evaluate the quality of pure to identify defects in coffee beans. This paper uses a dataset of
coffee using image processing methods. The processing 480 images, including defect classes such as black particles,
stages include image acquisition, preprocessing, segmen- broken particles, and holes, with 360 images used for training
tation, feature extraction, and classification. The classifi- and 120 images for testing. The results achieved were 93.3%
cation algorithms are Artificial Neural Networks (ANN), accuracy for individual object recognition and 75% accuracy
K-Nearest Neighbor (KNN) and Support Vector Machine for complex object recognition.
(SVM). Experimental results show that the ANN classifi- Meanwhile, the paper [26] presents a method to predict
cation algorithm achieves the highest accuracy of 89.45%, the types of defects in coffee beans by applying a Convolu-
compared to SVM (83.75%) and KNN (77.85%). Choosing tional Neural Network (CNN) model to images. This binary

VOLUME 12, 2024 126505

 H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

classification problem predicts suitable defect types in a given This paper [31] proposes an advanced multivariate strategy
coffee bean. The article proposes a CNN model to predict for near-infrared (HSI-NIR) hyperspectral image processing
coffee bean defects and achieves an accuracy of 90.44%. to automatically detect physical-chemical defects in green
Their model can be used systematically and effectively in the coffee. The study uses an object-based approach to charac-
coffee industry. terize individual coffee beans rather than individual pixels.
The paper [27] presents an approach to classify defective On these parameters, applying the k-NN (k-nearest neigh-
coffee beans in a short time and with high efficiency. The bors) classification algorithm and the prediction results on
tester is controlled by a Raspberry Pi 4 and equipped with test samples show that this method can distinguish defective
a camera to take photos and upload them to Google Drive. coffee beans free from defects and distinguish between dif-
Images were analyzed using three deep learning models: ferent types of defects.
Enhanced, ResNet-50, and AlexNet for training, validation, Previous studies have successfully applied machine learn-
and testing. The ResNet-50 model gave the best performance ing and deep learning algorithms to classify and evaluate
with an accuracy of 93.33%. That means it can accurately coffee quality. Image processing methods such as LBP, CNN,
classify defects and save working time sorting coffee beans. and YOLO have been effectively used to characterize and
The article [28] applies computer imaging to classify classify coffee bean defects. Some studies have combined
defects in green Arabica Gayo coffee beans by color, shape, near-infrared (NIR) hyperspectral sensor technology with
and textural characteristics. This paper aims to develop an deep learning to create high-precision, real-time error test-
image-based intelligent system to classify coffee bean quality ing systems. However, some studies have only focused on
based on the Indonesian National Standard. The K-Nearest classifying primary disabilities, not delving into more subtle
Neighbor (K-NN) and Support Vector Machine (SVM) meth- disabilities. While earlier algorithms could identify simple
ods are used with 13 parameters, including area, contrast, faults such as broken or black beans, detecting subtler or more
energy, correlation, homogeneity, roundness, perimeter, and complex defects was more challenging.
index color numbers. 1200 green Arabica coffee beans were Research gaps can be identified as follows. While earlier
captured using the Kinect V2 camera with training data algorithms could identify simple faults such as broken or
consisting of 1000 samples and testing data consisting of black beans, detecting subtler or more complex defects was
200 samples. more challenging. It is necessary to expand the datasets and
The paper [29] introduces a tool using the YOLO algorithm classify coffee defects in more detail to meet the actual needs
to select coffee quality, addressing the issue of manual of the coffee industry.
inspection being affected by external factors like fatigue and In addition, implementing computer vision systems for
lighting. The system uses a camera to capture images of coffee bean grading can involve significant upfront costs,
coffee beans, processed by Raspberry Pi and displayed on including hardware, software development, and integration
a laptop screen. After 10 tests on 100 coffee beans, the with existing manufacturing lines. Research should focus
YOLO algorithm achieved an average detection accuracy of on developing more affordable hardware components specif-
76.54% for perfect beans, 73.40% for imperfect beans, and ically designed for coffee bean grading. This includes
1% for undetected beans. This method maintains a successful exploring low-cost solutions that can deliver reliable perfor-
detection accuracy level of 75%. mance without the high costs associated with current systems.
Pinto et al. [18] developed an automatic classification sys- Addressing these research gaps could lead to more affordable
tem for the types of defects in green coffee beans using a and practical solutions for implementing computer vision
convolutional neural network. First, they developed an image systems in coffee bean grading, making advanced technology
processing system to classify images of green coffee beans accessible to a broader range of producers, and enhancing the
into each type of defect. They used a deep convolutional overall efficiency and quality of the coffee industry.
neural network, a state-of-the-art machine learning technique,
to perform the image processing. The results showed that C. COFFEE BEANS CLASSIFICATION APPLICATION
they successfully classified coffee beans with defects with Coffee bean defects are crucial for ensuring food quality
an accuracy ranging from 72.4% to 98.7%, depending on the and safety in coffee production and processing. Common
type of defect. defects include damaged, old, wet, or pest-infested seeds.
García et al. [30] present the use of a computer vision Manual classification is time-consuming and inaccurate,
system to inspect the quality and defects of green coffee beans leading to research on automation applications using com-
by applying image processing and machine learning tech- puter vision and machine learning. Advanced applications
niques integrated with an Arduino Mega board. The k-nearest use super-resolution cameras and deep learning to classify
neighbor algorithm is used to determine the quality of coffee particle defects near-infrared spectrum and achieve accuracy
beans and the corresponding types of defects. The system of up to 90-95%. Paper [32] describes a study on developing
includes logic routines, image processing, and supervised a deep learning model for automatic coffee bean quality
learning algorithms programmed using MATLAB and then classification integrated into a mobile system. The authors
saved to the Arduino board. Statistical analysis shows that tested two CNN architectures to develop a coffee qual-
this process can accurately identify defects and quality. ity classification model, including VGG16 and ResNet-152.

126506 VOLUME 12, 2024

H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

The experimental results show that the ResNet-152 architec- the information. The coffee bean classification procedure
ture achieves the highest classification accuracy of 73.3% consists of three main stages: bean detection, feature extrac-
and can be integrated into a working mobile application. tion, and bean classification. The process is considered
However, in this research, the integration and design of successful when the features of the coffee beans captured in
the mobile application were described only in functional the recorded image correspond to stored defects or healthy
design. status profiles. The classification results are extracted and
Meanwhile, the paper [33] proposes a deep learning-based stored for further analysis. This research focuses on ensur-
computer vision system for assessing the quality of roasted ing the system’s flexibility to handle large data streams and
coffee beans. The paper proposes a deep learning model, seamlessly process data from cameras, enabling real-time
MobileNetV2, embedded on Android smartphones to auto- operation with speed and user-friendly functionality.
mate the process of coffee roasting, allowing on-demand,
portable quality assessment. The model, trained on a dataset B. RESEARCH MODEL
of coffee bean images, achieved high accuracy (over 97%) in Our system comprises two primary layers: AI ENGINE and
classifying roasting degrees. USER APP. Fig. 1 illustrates the proposed system architec-
The authors presented research on developing a mobile ture for the coffee bean classification system. To initiate the
app for coffee bean ripeness recognition using digital image classification process, employees can submit raw photos or
processing [34]. This study uses the HSV color space and the stream videos through the USER APP during each session.
K-Nearest Neighbor (KNN) classification method to identify The main objective of the front-end application is to cap-
the ripeness of coffee beans (ripe, semi-ripe, green). The ture video from the user’s device and transmit it via the
mobile application developed in this study has two main WebSocket Protocol and the API Gateway. Video streams
features: data training and real-time recognition. The test from various sources, such as a desktop or laptop camera,
results showed that the app achieved 95.56% accuracy with a a mobile app, or a website, are all sent to the API gateway.
best membership value (k) of 3. The remaining processes are centrally executed on the shared
There is a need to develop deep learning models that run server of the system to minimize the application’s footprint
efficiently and deliver fast, accurate results on mobile devices and ensure efficient image processing.
with limited resources. Additionally, the application must
feature a simple, user-friendly interface suitable for mobile
devices. Addressing these gaps will lead to mobile applica-
tions that effectively classify coffee bean defects, are easy to
use, and can be widely adopted by small and medium-sized
coffee enterprises (SMEs). This proposed research aims to
address these identified gaps.

III. PROPOSED RESEARCH MODEL AND EXPERIMENT

SETUP
A. RESEARCH METHODOLOGY
This research presents an approach to enhance the efficiency
of the green coffee bean classification system. This paper
describes the methods and techniques for constructing arti-
ficial vision systems used in the quality inspection of green
coffee beans. It covers the necessary software, hardware,
and imaging techniques required for the system. A typical
machine vision system comprises an image acquisition stage,
an image processing stage, and statistical analysis. Develop-
ing a machine vision system enables automation of quality
and defect inspection, eliminating the need for manual and
repetitive work traditionally performed in the industry. This FIGURE 1. Proposed prototype coffee bean defect classification
involves image acquisition, image processing, and the imple- application.
mentation of a learning algorithm, which are explained in
detail below. The coffee bean images or videos will be transferred to the
The proposed model integrates image classification tech- AI Engine layer. The AI Engine procedure involves inputting
nologies and real-time ingestion of streaming media sources. the raw image into the program to analyze, interpret the
Our system accepts input data from various sources, includ- subject structure, feature extraction, and classification.
ing camera images and videos captured on mobile, desktop, The first step in the process involves image acquisition,
or laptop devices. Before the classification process, the input wherein electronic signals are captured from a sensor and
data undergoes pre-processing to normalize and standardize converted into a numerical representation using a camera or

VOLUME 12, 2024 126507

 H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

similar device. The quality of the acquired image is directly

affected by the lighting conditions during the acquisition
phase. It is crucial to ensure uniform and non-specular illu-
mination, particularly when extracting key features related to
coffee bean quality, such as color and the presence of external
defects.
Subsequently, image processing encompasses various
tasks aimed at manipulating the digital images obtained.
This stage can be divided into two parts: pre-processing
and image processing. Pre-processing includes adjustments
such as grayscale conversion, focus correction, contrast or
sharpness enhancement, and noise reduction. The primary
objective of pre-processing is to enhance the image qual-
ity and prepare it for subsequent processing steps. On the
other hand, image processing involves tasks such as object
segmentation and feature extraction. Multiple features can
be extracted from the objects of interest, including area,
perimeter, length, shape, and color. The selection of appro-
priate features depends on the specific samples used and the
intended purpose of the machine vision system.
Finally, classification refers to the process of assigning a
category to a new observation. This is achieved by comparing
the new observation with a training set of data or observations FIGURE 2. Coffee bean defects classification procedure.

with known categories. This task is often referred to as pattern

recognition or learning algorithms. In this paper, we uti- adjusted to ensure they enclose objects accurately. Once the
lize the YolovV8 model to develop a coffee classification labeled data reached a sufficient size and level of accuracy,
algorithm. we labeled the dataset for training.
The classification data will be stored in the Production In the training model, we train our model with Yolov8.
Database Integration and shown in a dashboard format to Yolov8 is a deep learning model that can run efficiently,
facilitate decision-making by the Head or Manager. giving fast and accurate results on mobile devices with lim-
ited resources. YOLO (You Only Look Once) is a widely
C. COFFEE BEAN DEFECTS CLASSIFICATION PROCEDURE recognized object detection algorithm that has significantly
The coffee bean images or videos will be transferred to the transformed the domain of computer vision [35]. YOLO
AI Engine layer. The AI Engine procedure involves inputting comprises a combination of convolutional layers, which facil-
the raw image into the program to analyze and interpret the itate the extraction of image features, and connected layers,
subject structure, feature extraction, and classification. which predict the probability and coordinates of the detected
The schematic representation of the proposed procedure object [35]. YOLO utilizes the grid size of 7 × 7 to partition
for coffee bean classification is illustrated in Fig. 2. We divide the image. It then predicts the presence of an object within
our data into three sub-datasets: 80% for training data, 15% each cell, along with its size, confidence score, and center
for testing data, and 5% for validation data. In the label- point location. The task assigned to each cell is to predict two
ing phase, after receiving data from the factory specialists, bounding boxes describing the rectangular region around an
the image data is labeled using built-in features such as object. YOLOv8, the eight-generation version, is the current
bounding boxes, particle localization, or labeling tools. In our iteration in use. In addition, we employed the Stochastic Gra-
research, the bounding-box technique was employed to anno- dient Descent (SGD) optimizer with momentum throughout
tate images of coffee beans by drawing rectangular boxes the training of our YOLOv8 model. The model’s weights are
around each bean or defect. These boxes are defined by updated iteratively by computing the gradient of the loss func-
four coordinates representing the corners of the rectangle, tion for each parameter and thereafter moving in the direction
providing precise information about the location and size opposite to the gradient. By including momentum, the opti-
of the objects within the images. Each bounding box also mizer’s convergence is expedited since it mitigates abrupt
includes a class label indicating the enclosed defect or object updates, preventing the model from becoming trapped in
type. After the dataset was labeled with these bounding boxes, local minima and resulting in swifter convergence. Once the
it was further augmented through techniques like rotation, library and variables are initialized, the YOLOv8 algorithm
flipping, and scaling to increase the diversity of the training model is executed. Users can select images, stream live video
data. This augmentation improves the model’s robustness from a camera, or opt for image and video files stored on
and generalization by exposing it to various scenarios. When their devices. The processing occurs within a while loop,
these transformations are applied, the bounding boxes are where each frame is captured and processed. In the context of

126508 VOLUME 12, 2024

H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

YOLOv8, the model can further analyze and classify objects server. The secondary server houses read-only copies of zone
once objects are detected and localized within an image or files, which are inaccessible for modification. The computer
video frame. After successfully identifying the bounding box specifications are presented as below:
coordinates of the detected objects, YOLOv8 can extract the • Operating System: Ubuntu 20.04.5 LTS
corresponding regions of interest (ROI) for each object. These • Processor: AMD® Ryzen 7 5800 × 8-core processor
ROIs are then inputted into a classification network or classi- × 16
fier, typically a deep neural network such as a convolutional • Memory: 128.0 GB RAM
neural network, which has been trained for specific classifi- • Graphics: NVIDIA Corporation TU106 [GeForce
cation tasks. The classifier examines the extracted regions of RTX 2060 Rev. A]
interest and assigns class labels to the objects. By integrating The research utilizes Microsoft’s Visual Studio as a pro-
object detection and classification techniques, YOLOv8 pro- totyping and programming software for AI functions and
vides a comprehensive approach to accurately localizing and main programming. It supports Python programming in the
categorizing objects in images or video frames. YOLOv8 uses Coffee Detection and Classification Module and Node.js,
its trained weights to predict bounding boxes and class labels React Native in web-app and mobile applications. The soft-
for new, unseen images during inference. The model outputs ware enables development through a container via a remote
a set of bounding boxes with associated confidence scores for Linux Subsystem and provides class libraries for face recog-
each detected object. This approach allows YOLOv8 to detect nition algorithms. PostgreSQL is employed as the database
and classify objects, like coffee bean defects, even in complex management engine, facilitating the creation of a relational
and varied scenes, making it a powerful tool for real-world database for registration, user, face, and transaction data.
applications. The data is processed, extracted, and visualized in the
Once the model has been trained, we proceed to integrate application’s interface through the database connection API
the YOLOv8 coffee bean categorization model into our appli- mechanism.
cation. We construct an image classification function using The mobile application development involves using Expo.
the classification library. The function takes an image to This framework allows for multi-platform development,
be recognized as input. The classification function produces reducing the recompile cost of Native by utilizing Hot-
outputs that include information about the identified objects Loading, enabling developers to trace back their code quickly
and their respective positions within the image. WebSocket is and intuitively without spending excessive time building and
employed as a communication protocol facilitating real-time, running the application [36].
bidirectional interaction between a client and a server through In our experiment, real-time processing was a critical com-
a singular, persistent connection. Within the framework of ponent, and the choice of camera played a significant role.
React Native, the WebSocket protocol can be employed to The experiments were conducted using the built-in cameras
create a connection with a WebSocket server and facilitate of two mobile phones, the iPhone 8 and the Samsung Galaxy
the exchange of data between the application and the server. S8. The specifications of these cameras are as follows:
The WebSocket API facilitates the creation of WebSocket iPhone 8:
instances, the transmission and reception of messages, the
• Camera Resolution: 12 MP
handling of events, and the management of the WebSocket
• Frame Rate: Up to 60 fps
connection.
• Additional Features: Optical image stabilization, wide-
angle lens
D. RESEARCH ENVIRONMENT
Empirical research shows challenges small and medium-sized Samsung Galaxy S8:
enterprises (SMEs) face when allocating resources for IT • Camera Resolution: 12 MP
infrastructure investments. A well-designed infrastructure • Frame Rate: Up to 30 fps
utilizing open-source software or cloud services can help • Additional Features: Dual pixel autofocus, wide-angle
reduce the initial financial investment. This study aims to lens
provide resources for implementing suitable methods for These cameras were chosen for their capability to provide
SMEs in the coffee industry and small-scale agriculturalists, high-resolution images and adequate frame rates essential for
addressing these challenges. effective real-time processing and accurate classification of
In our research paper, the computer served as the primary coffee bean defects. The integration of these mobile phone
hardware component in our system, facilitating various tasks cameras with our deep learning model was optimized to
such as preprocessing raw images, extracting image features, ensure that the real-time performance met the required stan-
constructing recognition algorithms, and developing a mobile dards for practical application.
application. Additionally, we designed our computer to func-
tion as the primary server for the system’s process flows. The IV. IMPLEMENT RESULT
primary server is responsible for storing all relevant resource A. RESEARCH DATASET
records and managing all queries related to a specific domain. The sample used in this study consisted of commercially
A supplementary server is also present in the form of a cloud traded Robusta coffee green beans, specifically harvested

VOLUME 12, 2024 126509

 H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

in the year 2021. These beans were obtained from Tuan

Loc Commodities, a company based in Ho Chi Minh City,
Vietnam, specializing in the processing and exporting of
green coffee [37]. During the field survey at Tuan Loc Com-
modities, the coffee sorting process was done manually by
spreading the coffee beans on a flat white surface. Then,
workers sorted them by comparing them to the company’s
product standards.
Deep learning classification models require substantial
training data and validation and testing data to improve accu-
racy and speed. We opted to generate a proprietary dataset
to evaluate the real-time performance of the deep learning
engine on our green coffee bean samples. In our experiment,
we aimed to create a controlled environment for coffee bean
classification by using a uniform white surface (represented
by an A4 sheet of paper) as a symbolic surface. This approach
allowed us to focus on the classification process while ensur-
ing consistency in our dataset. We recognize that coffee beans
would not be arranged on paper in an industrial setting,
and the environment would be more complex. To address
this, we varied the conditions under which the images were
captured, including different lighting conditions, angles, and
settings, to simulate a more realistic scenario and make our
dataset more robust. We conducted experiments to capture
coffee beans under various settings, including variations in
illumination size, different light conditions, and angles. After-
ward, we standardized all the raw photos. We applied labels to
the images during the pre-processing stage using a bounding FIGURE 3. Sample of coffee bean defects dataset.
box technique, which involves enclosing the digital image
with a border. This step was performed to create our training
dataset. Throughout our investigation, we generated a dataset Application was developed using the React Native framework
of 4640 photos, encompassing 9 different defect categories in conjunction with the Expo development tool. Fig. 4 illus-
in coffee bean defects. We describe our dataset and sample trates the initiation of the Expo React Native by executing
images in Fig. 3 below. the Yarn command. The provided QR Code can be used
Deep learning recognition models require extensive train- to streamline the installation and execution process of the
ing, validated, and testing data for accuracy and speed application.
enhancement. This paper aims to create a real-time dataset for
AI attendance evaluation, utilizing available online datasets.
The dataset involves capturing a bean under various light
conditions and angles, cropping the bean in each image, and
standardizing all raw images. A bounding box is used to
create the training dataset. An automatic function is built
to extract bean data from the camera of a mobile applica-
tion, allowing users to take their photos. The input image is
then fed into bean detection models, which are presented in
the next section. If the detection module’s accuracy exceeds
85%, the bordered image is sent to the output folder, saving
time and generating research datasets within the application’s
operating time. FIGURE 4. Run the expo react native by yarn start block.

B. COFFEE BEAN DEFECTS CLASSIFICATION APPLICATION After scanning the QR code, Expo will proceed to build the
RESULTS JavaScript bundle. Users simply need to wait for the instal-
Integrating a mobile application into the coffee bean classi- lation application to complete. Fig. 5 showcases the main
fication system offers users more convenience, as it allows interface of the application. To minimize time consumption
for flexible use in various working contexts such as farms, during the login process and complex tasks, we have inten-
stock areas, and processing plants at any time. The Mobile tionally designed the interface to be simple and user-friendly.

126510 VOLUME 12, 2024

H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

It includes two distinct buttons for the live camera and

image upload functionalities. Employees can easily access
and review historical records by logging into the web version.
Furthermore, the primary interface also offers the capability
to initiate conversations to request information or assistance.

FIGURE 6. Live camera function.

FIGURE 5. Main home screen.

The Live Camera function allows users to capture images

or record vieos within the application using the camera on
their phone or device (refer to Fig. 6). The application is
equipped with the capability to accurately classify and iden-
tify nine common categories of faults found in coffee beans,
offering the advantage of real-time detection. These potential
defects include issues such as Mold, Mixed, Broken, Big
Broken, Black, Partial Black, Brown, Insect, and Immatures.
Each specific defect type is highlighted by distinct frames of
various colors. Fig. 6 provides an example of the Black defect,
indicated by a green outline.
The Upload Images function enables users to select one or
multiple images. The Upload Images feature of the applica-
tion allows users to select and upload individual or multiple
images from their device’s storage or gallery. This function-
ality offers users the freedom and convenience to choose
the relevant images they want to utilize or examine within
the application. Once the user has selected the images, the FIGURE 7. Upload images function.
application processes and manages them accordingly. The
Upload Images feature is particularly advantageous when
users have pre-existing photographs or need to study multiple A zoom function has been developed for processed pho-
images concurrently. This feature optimizes the workflow and tos, shown in Fig. 8. The Zoom feature within the defect
saves users time and effort by enabling batch processing. detection application enables users to enhance the amount
Each specific defect type is highlighted by distinct frames of magnification for presented images. This feature proves
of various colors and shown in Processed Images. We also advantageous when users see a particular region or concen-
provide a comparison view of original and processed images trate on an image display. By expanding the visible portion
for users to easily track and read the results. Fig. 7 provides of the image, users can see a greater extent of the image and
an example of the Upload Images function. discern patterns or contextual details of the identified flaws.

VOLUME 12, 2024 126511

 H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

Including the Zoom feature in the program boosts its usability

and versatility. This feature empowers users by granting them
control over the amount of zooming, enabling them to inspect
the photographs thoroughly.

FIGURE 9. Coffee bean defects classification screen.

as shown in Fig. 10. Precision measures the accuracy of iden-

tifying positive instances. A higher value indicates a more
accurate model. Recall is an important metric as it quantifies
the ability to predict positive cases correctly about the total
number of samples. A higher Recall value suggests a reduced
chance of missing positive instances. The F1 score serves as
a balanced evaluation metric, considering both Recall and
Precision. It provides a basis for model selection, with a
higher F1 score indicating more accurate results from the
model.
Fig. 10 (a) illustrates the relationship between the
FIGURE 8. Defect image online viewing. F1-score and the increment in confidence throughout the
training process. The model achieved a cumulative F1 score
The results of identifying various types of defects are of 38%. Fig. 10 (b) displays the precision-recall curves (PR
also depicted in Fig. 9. The classification of defects into curves). The graphical representations illustrate the relation-
specific defect types based on different color box types is ship between precision levels and recall values throughout the
shown. Additionally, particle counts per defect are presented training process. The results indicate that the model achieved
and stored in a centralized production database. This data a mean average precision ([email protected]) of 38% across all pave-
will undergo further analysis and will be utilized for more ment classes. The curves exhibit proximity and concentration
comprehensive forecasting purposes. in the upper right quadrant, suggesting that the model pos-
sesses a high level of accuracy in predicting and classifying
V. PERFORMANCE AND EVALUATION distresses.
Finally, the performance of the coffee bean defect classi- The improvement in time classifies coffee beans even if
fication system is assessed through experimental analysis. mold or insect damage is significant. The system can help to
The model used in the research consists of 225 layers, with classify 9 types of coffee bean defects has been substantially
a total of 3,012,798 parameters and 3,012,782 gradients. expedited, with the time required for this task reduced to a
The computational complexity of the model is measured at mere 1 to 3 seconds. Our system can help the organization
8.2 GFLOPs. We train our model in 300 epochs. The decision without the reliance on stationary equipment, especially in
to terminate the training process prematurely was made due the framework of the distributed working model or in a large
to the absence of any discernible improvement in the last working environment.
50 epochs. The most favorable outcomes were seen during To evaluate the performance of our system, we prepared
epoch 84, and the model that yielded these findings was saved Table 1 to present our comparison of existing coffee bean
as the best model in this instance. A total of 134 epochs were defect classification applications in several key factors. Accu-
completed within a timeframe of 1.070 hours. racy varies significantly across systems, with the Coffee
The primary performance evaluation metrics of the clas- Fruit Classification model achieving the highest accuracy
sification model include Precision, Recall, and F1 Score, at 95.56%, though it focuses on identifying the ripeness

126512 VOLUME 12, 2024

H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

TABLE 1. Comparison of existing coffee bean defects classification

applications.

FIGURE 10. Performance evaluation metrics.

of coffee fruits rather than defects in processed beans. The

Coffee Bean Inspection Machine stands out in industrial
settings with a 93.3% accuracy using the ResNet-50 model,
demonstrating suitability for high-volume production envi-
ronments. Mobile applications like Cafeon and the Coffee
Quality Assessment System offer more accessible solutions
with a 73.3% accuracy but may sacrifice precision compared
to industrial counterparts. The ability to detect multiple defect
types is critical, as seen in the Coffee Bean Selector, which
identifies four defect types with an average accuracy of
around 75%. Ours advances this further by detecting nine
different defect types, though it achieves a lower average
accuracy of 38% using the YOLOV8 model. Our system’s
real-time detection capability, with processing times between
1 to 3 seconds, is particularly advantageous for mobile and
cloud-based applications where quick feedback is essential.
The technological approaches also vary, from Raspberry
Pi-based systems to cloud-based architectures, highlighting
the trade-offs between scalability, speed, and accuracy.
Our system offers several significant advantages, partic-
ularly in its ability to perform real-time processing, with
detection times ranging from just 1 to 3 seconds. This
quick response is essential in dynamic environments where

VOLUME 12, 2024 126513

 H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

TABLE 1. (Continued.) Comparison of existing coffee bean defects using YOLOv8. Additionally, a software application has
classification applications.
been developed to enable the real-time identification of cof-
fee beans through video and image streaming. WebSocket
is utilized for communication purposes. The implemented
technology has demonstrated the capability to detect and
distinguish nine different types of coffee beans, even in cases
involving mold or insect damage. As a result, the system’s
efficiency has significantly improved, with the required iden-
tification time reduced to 1 to 3 seconds. Its cost-effective and
scalable infrastructure, achieved through a cloud-based solu-
tion paired with a low-cost primary server, further enhances
its accessibility, allowing for easy scaling of operations with-
out substantial hardware investments.
Integrating this deep learning model into a mobile appli-
cation adds convenience, enabling users to conduct quality
assessments in various settings, from production facilities
to remote farms. Moreover, cloud technology ensures the
system is future-proof, allowing for seamless updates and
immediate feedback is crucial for effective quality control. integration of new models or improvements. This solution
Additionally, the system’s versatility is evident in its ability holds the potential to assist organizations in reducing their
to detect nine different types of coffee bean defects, mak- reliance on fixed equipment, particularly in remote work
ing it suitable for comprehensive quality assessments. Its models or large work environments. By leveraging computer
cost-effective and scalable infrastructure, achieved through a vision and artificial intelligence, the coffee industry could
cloud-based solution paired with a low-cost primary server, achieve enhanced efficiency and accuracy in the bean clas-
further enhances its accessibility, allowing for easy scaling sification process, contributing to quality improvement and
of operations without substantial hardware investments. The sustainable growth.
integration of this deep learning model into a mobile appli- Future work can be defined as improving model accu-
cation adds convenience, enabling users to conduct quality racy, which can be achieved by incorporating more advanced
assessments in various settings, from production facilities deep learning architectures, fine-tuning hyperparameters, and
to remote farms. Moreover, cloud technology ensures the leveraging larger, more diverse datasets. Next, expanding the
system is future-proof, allowing for seamless updates and range of detectable defects includes identifying a broader
integration of new models or improvements. spectrum of defects, such as moisture content irregularities
However, using the YOLOv8 model to develop a mobile or internal damage. Our system can also be taken advantage
application to classify coffee bean defects also has some of by integrating multi-modal data, such as combining visual
limitations. First, YOLOv8 is designed for general object data with spectral imaging or sensor inputs, which can pro-
detection and classification, but further tuning is still needed vide a more comprehensive assessment and detect defects that
to detect and classify various types of coffee bean defects and are not visually apparent. Additionally, optimizing real-time
improve model accuracy. Integrating other sensors, such as performance is important, and exploring lightweight model
cameras, electronic scales, and light sensors, can significantly architectures or edge computing could further reduce pro-
improve the system’s accuracy. It is necessary to research data cessing times, making the system suitable for high-speed
fusion techniques to maximize information from these sensor production lines. Finally, conducting extensive field tests
sources. Finally, it is necessary to design a user-friendly and refining deployment strategies to improve the system’s
interface that is easy to use on mobile devices while integrat- robustness and practical applicability in various real-world
ing user support features such as instructions, feedback, and environments. Furthermore, adapting the methodology for
interactivity. other agricultural products could extend the system’s benefits
beyond coffee, while integrating with IoT devices and smart
VI. CONCLUSION agriculture platforms could enhance real-time monitoring and
In conclusion, the coffee industry plays a crucial role automation in agricultural quality control.
in economic transformation and sustainable development.
To optimize coffee quality, farmers need to select coffee AVAILABILITY OF DATA AND MATERIALS
beans before mass production carefully. However, tradi- Please contact the corresponding author for data requests.
tional eye assessment methods are both time-consuming The coding environment employed is Visual Studio, encom-
and prone to inaccuracies. This paper aims to automate the passing Python programming capabilities alongside Node.js
classification process using computer vision and artificial and React Native. The database management system in use
intelligence techniques. Image processing and data augmen- is PostgreSQL. Developing mobile applications uses Expo
tation methodologies are employed, and a model is trained as a framework for building React Native applications. The

126514 VOLUME 12, 2024

H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

training data and testing data are published [38]. Developing [13] M. N. S. Akbar, E. Rachmawati, and F. Sthevanie, ‘‘Visual feature and
mobile applications uses Expo as a framework for building machine learning approach for arabica green coffee beans grade deter-
mination,’’ in Proc. 6th Int. Conf. Commun. Inf. Process., Nov. 2020,
React Native applications. We have published the resources pp. 97–104.
of the article on GitHub [39] and an instruction for our [14] J. W. Mugwe, ‘‘Factors influencing coffee production by small scale
Marine Equipment E-commerce Data Hub Ecosystem (Video farmers: A case of Tetu constituency, Kenya,’’ Master Arts Project Planning
Manag., Fac. Educ., Univ. Nairobi, Kenya, 2014.
Abstract) [40]. [15] M. Bozzola, S. Charles, T. Ferretti, E. Gerakari, H. Manson, N. Rosser,
[38] Hong Danh Thai, Han-Jong Ko, Jun-Ho Huh, ‘‘Cof- and P. von der Goltz, The Coffee Guide. Geneva, Switzerland: International
fee Beans’’, [Online]. Available: https://www.kaggle.com/ Trade Centre, 2021.
datasets/hongdanhthai/coffeebeans [16] SCAA Green Coffee Beans Classification. [Online]. Available:
https://www.coffeeresearch.org/coffee/scaaclass.htm
[39] Hong Danh Thai, Han-Jong Ko, Jun-Ho Huh, [17] R. G. Apaza, C. E. Portugal-Zambrano, J. C. Gutiérrez-Cáceres, and
‘‘Coffee-Beans-Real-Time-Classification’’, [Online]. Avail- C. A. Beltrán-Castañón, ‘‘An approach for improve the recognition of
able: https://github.com/thaihongdanhh/Coffee-Beans-Real- defects in coffee beans using retinex algorithms,’’ in Proc. XL Latin Amer.
Comput. Conf. (CLEI), Sep. 2014, pp. 1–9.
Time-Classification [18] C. Pinto, J. Furukawa, H. Fukai, and S. Tamura, ‘‘Classification of green
[40] Hong Danh Thai, Han-Jong Ko, Jun-Ho Huh, Video coffee bean images basec on defect types using convolutional neural net-
Abstract ‘‘Coffee Beans Real-Time Classification’’, [Online]. work (CNN),’’ in Proc. Int. Conf. Adv. Informat., Concepts, Theory, Appl.
(ICAICTA), Aug. 2017, pp. 1–5, doi: 10.1109/ICAICTA.2017.8090980.
Available: https://www.youtube.com/watch?v=WvQ4jbGQ
[19] P. Oliveri, C. Malegori, M. Casale, E. Tartacca, and G. Salvatori, ‘‘An inno-
WB4 vative multivariate strategy for HSI-NIR images to automatically detect
defects in green coffee,’’ Talanta, vol. 199, pp. 270–276, Jul. 2019.
REFERENCES [20] M. Bedaso, M. Meshesha, and C. Diriba, ‘‘Comparing performance of
classification algorithms to use for grading coffee’s raw quality by using
[1] M. Adhikari, E. L. Isaac, R. R. M. Paterson, and M. A. Maslin, ‘‘A review image processing techniques,’’ AGBIR, vol. 39, no. 2, pp. 491–495, 2023.
of potential impacts of climate change on coffee cultivation and mycotox- [21] M. M. Hidayat, E. Prasetyo, and B. T. Wicaksono, ‘‘Classification of
igenic fungi,’’ Microorganisms, vol. 8, no. 10, p. 1625, Oct. 2020. roasting rates in coffee beans by digital image processing using the Naive
[2] J. A. Barreto Peixoto, J. F. Silva, M. B. P. P. Oliveira, and R. C. Alves, Bayes classifier (NBC) method,’’ Jurnal IPTEK, vol. 27, no. 1, pp. 23–30,
‘‘Sustainability issues along the coffee chain: From the field to the cup,’’ May 2023.
Comprehensive Rev. Food Sci. Food Saf., vol. 22, no. 1, pp. 287–332,
[22] E. Arboleda, ‘‘Classification of immature and mature coffee beans using
Jan. 2023.
texture features and medium K nearest neighbor,’’ J. Artif. Intell. Technol.,
[3] B. Lewin, D. Giovannucci, and P. Varangis, ‘‘Coffee markets: New vol. 3, no. 3, pp. 114–118, May 2023.
paradigms in global supply and demand,’’ World Bank Agricult. Rural
[23] V. A. M. Luis, M. V. T. Quiñones, and A. N. Yumang, ‘‘Classification of
Develop. Discuss. Paper, vol. 1, no. 3, pp. 1–150, Mar. 2004.
defects in robusta green coffee beans using YOLO,’’ in Proc. IEEE Int.
[4] R. J. Clarke, ‘‘Green coffee processing,’’ in Coffee: Botany, Biochemistry
Conf. Artif. Intell. Eng. Technol. (IICAIET), Sep. 2022, pp. 1–6.
and Production of Beans and Beverage. Boston, MA, USA: Springer, 1985,
[24] S.-Y. Chen, M.-F. Chiu, and X.-W. Zou, ‘‘Real-time defect inspection of
pp. 230–250.
green coffee beans using NIR snapshot hyperspectral imaging,’’ Comput.
[5] N.-F. Huang, D.-L. Chou, and C.-A. Lee, ‘‘Real-time classification of green
Electron. Agricult., vol. 197, Jun. 2022, Art. no. 106970.
coffee beans by using a convolutional neural network,’’ in Proc. 3rd Int.
Conf. Imag., Signal Process. Commun. (ICISPC), Singapore, Jul. 2019, [25] T. A. Heryanto and I. G. B. B. Nugraha, ‘‘Classification of coffee
pp. 107–111, doi: 10.1109/ICISPC.2019.8935644. beans defect using mask region-based convolutional neural network,’’ in
Proc. Int. Conf. Inf. Technol. Syst. Innov. (ICITSI), Bandung, Indonesia,
[6] H. L. Gope and H. Fukai, ‘‘Normal and peaberry coffee beans classifica-
Nov. 2022, pp. 333–339, doi: 10.1109/ICITSI56531.2022.9970890.
tion from green coffee bean images using convolutional neural networks
and support vector machine,’’ Int. J. Comput. Inf. Eng., vol. 14, no. 6, [26] J.-Y. Lee and Y.-S. Jeong, ‘‘Prediction of defect coffee beans using CNN,’’
pp. 189–196, 2020. in Proc. IEEE Int. Conf. Big Data Smart Comput. (BigComp), Jan. 2022,
pp. 202–205, doi: 10.1109/BigComp54360.2022.00046.
[7] J. P. Rodríguez, D. C. Corrales, J.-N. Aubertot, and J. C. Corrales, ‘‘A com-
puter vision system for automatic cherry beans detection on coffee trees,’’ [27] T. Micaraseth, K. Pornpipatsakul, R. Chancharoen, and G. Phanomchoeng,
Pattern Recognit. Lett., vol. 136, pp. 142–153, Aug. 2020. ‘‘Coffee bean inspection machine with deep learning classification,’’ in
Proc. Int. Conf. Electr., Comput., Commun. Mechatronics Eng. (ICEC-
[8] Y. Hendrawan, B. Rohmatulloh, F. I. Ilmi, M. R. Fauzy, R. Damayanti,
CME), Maldives, Maldives, Nov. 2022, pp. 1–5, doi: 10.1109/ICEC-
D. F. Al Riza, M. B. Hermanto, and Sandra, ‘‘AlexNet convolutional neural
CME55909.2022.9987835.
network to classify the types of Indonesian coffee beans,’’ IOP Conf. Ser.,
Earth Environ. Sci., vol. 905, no. 1, Nov. 2021, Art. no. 012059. [28] A. Maghfirah and I. S. Nasution, ‘‘Application of colour, shape, and texture
[9] X. Hu, W. Liu, C. Liu, and Y. Qian, ‘‘Rapid identification of producing parameters for classifying the defect of Gayo Arabica green coffee bean
area of coffee bean based on terahertz spectroscopy and support vector using computer vision,’’ IOP Conf. Ser., Earth Environ. Sci., vol. 951, no. 1,
machine,’’ Trans. Chin. Soc. Agricult. Eng., vol. 33, no. 9, pp. 302–307, 2022, Art. no. 012097.
2017. [29] R. Putri and R. Tiara, ‘‘Design and development of a coffee bean selector
[10] M. V. C. Caya, R. G. Maramba, J. S. D. Mendoza, and P. S. Suman, using the YOLO algorithm,’’ Int. J. Eng., Sci. Inf. Technol., vol. 2, no. 4,
‘‘Characterization and classification of coffee bean types using pp. 84–89, Dec. 2022.
support vector machine,’’ in Proc. IEEE 12th Int. Conf. Humanoid, [30] M. García, J. E. Candelo-Becerra, and F. Hoyos, ‘‘Quality and defect
Nanotechnol., Inf. Technol., Commun. Control, Environ., Manage. inspection of green coffee beans using a computer vision system,’’ Appl.
(HNICEM), Manila, Philippines, Dec. 2020, pp. 1–6, doi: Sci., vol. 9, no. 19, p. 4195, Oct. 2019.
10.1109/HNICEM51456.2020.9400144. [31] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, ‘‘You only look once:
[11] R. H. Montes Condori, J. H. Chuctaya Humari, C. E. Portugal-Zambrano, Unified, real-time object detection,’’ in Proc. IEEE Conf. Comput. Vis.
J. C. Gutiérrez-Cáceres, and C. A. Beltrán-Castañón, ‘‘Automatic classifi- Pattern Recognit. (CVPR), Jun. 2016, pp. 779–788.
cation of physical defects in green coffee beans using CGLCM and SVM,’’ [32] R. Janandi and T. W. Cenggoro, ‘‘An implementation of convolutional neu-
in Proc. XL Latin Amer. Comput. Conf. (CLEI), Montevideo, Uruguay, ral network for coffee beans quality classification in a mobile information
Sep. 2014, pp. 1–9, doi: 10.1109/CLEI.2014.6965169. system,’’ in Proc. Int. Conf. Inf. Manage. Technol. (ICIMTech), Aug. 2020,
[12] E. R. Arboleda, A. C. Fajardo, and R. P. Medina, ‘‘Classification pp. 218–222.
of coffee bean species using image processing, artificial neural net- [33] M. Hakim, T. Djatna, and I. Yuliasih, ‘‘Deep learning for roasting coffee
work and k nearest neighbors,’’ in Proc. IEEE Int. Conf. Innov. bean quality assessment using computer vision in mobile environment,’’
Res. Develop. (ICIRD), Bangkok, Thailand, May 2018, pp. 1–5, doi: in Proc. Int. Conf. Adv. Comput. Sci. Inf. Syst. (ICACSIS), Oct. 2020,
10.1109/ICIRD.2018.8376326. pp. 363–370, doi: 10.1109/ICACSIS51025.2020.9263224.

VOLUME 12, 2024 126515

 H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

[34] O. Sudana, D. Witarsyah, A. Putra, and S. Raharja, ‘‘Mobile application University, from September 2014 to August 2016. And, he was an Asso-
for identification of coffee fruit maturity using digital image processing,’’ ciate Professor (Tenure Track) with the Department of Agricultural Science,
Int. J. Adv. Sci., Eng. Inf. Technol., vol. 10, no. 3, pp. 980–986, Jun. 2020. Korea National Open University, from September 2016 to August 2024.
[35] J. Kosalos, R. Stephen, S. Diaz, P. Songer, and M. Alves, SCAA Arabica From 2008 to 2014, he was the Chief Scientist of Jeju Special Self-Governing
Green Coffee Defect Handbook, 2nd ed., Irvine, CA, USA: Specialty Province. From 2005 to 2007, he was a Visiting Researcher with Kyoto
Coffee Association of America, 2013, pp. 1–32. University, Japan. From 2004 to 2005, he was with Korea Rural Development
[36] Tuan Loc Commodities Co. (Nov. 12, 2023). About Tuan Loc Commodi- Administration in Suwon, eventually as a Senior Agriculture Scientist. His
ties. [Online]. Available: https://tlcommodities.com/ research interests include smart animal farm livestock housing environment
[37] D. Ward, React Native Cookbook: Recipes for Solving Common React
control and precision feeding management. Much of his work has been
Native Development Problems. Birmingham, U.K.: Packt Publishing,
on improving the understanding, design, and performance of animal waste
2019.
[38] H. D. Thai, H.-J. Ko, and J.-H. Huh. Coffee Beans. Accessed: treatment and utilization and smart livestock systems, mainly through the
Oct. 15, 2023. [Online]. Available: https://www.kaggle.com/datasets/ application of sensing, data collection, statistics, and performance evalua-
hongdanhthai/coffeebeans tion. In the dust generation on livestock area, he has worked a study on
[39] H. D. Thai, H.-J. Ko, and J.-H. Huh. Coffee-Beans-Real-Time- the characteristics of fine dust generated in the agricultural and livestock
Classification. Accessed: Oct. 15, 2023. [Online]. Available: sector. In addition, he had made the Korean Smart Livestock Farm Model
https://github.com/thaihongdanhh/Coffee-Beans-Real-Time- as the Chairperson of Subcommittee on Kosfarm. He is the co-author of
Classification Agriculture and Livestock Environment (Korea National University Press,
[40] H. D. Thai, H.-J. Ko, and J.-H. Huh. Coffee Beans Real-Time Classi- 2018) and Smart Farming (Minjung Publisher, 2020); and is the author
fication. Video Abstract. Accessed: Oct. 15, 2023. [Online]. Available: of over 80 articles on agriculture and livestock environment. He holds six
https://www.youtube.com/watch?v=WvQ4jbGQWB4 patents deriving from his research. He has given numerous invited talks
and tutorials and is a consultant to producer’s organizations involved in
agricultural environmental engineering.
Prof. Ko is an Editor of Asian-Australasian Association of Animal
Production Societies (AAAP) Review, Korean Society of Odor Research
HONG-DANH THAI was born in Ho Chi and Engineering, and Korean Society of Environmental Health. He was
Minh City, Vietnam. He received the degree in the Chair of the Resource Organization of the Ministry of Agriculture,
information system engineering from the Posts Food and Rural Affairs of Republic of Korea. Also, he received the
and Telecommunications Institute of Technology, Young Scientist Award in Korea Rural Development Administration and
Ho Chi Minh City, in 2015, and the M.S. degree in won a Korea Government Award. For more information visit the link
data informatics from National Korea Maritime & (https://scholar.google.co.kr/citations?user=shwv2iEAAAAJ).
Ocean University, Republic of Korea, where he
is currently pursuing the Ph.D. degree with the
Department of Data Informatics.
Since September 2022, he has been with the Join
Global R&E Program for Interdisciplinary Technologies of Ocean Renew-
able Energy (BK 21 Four Research Group) of Interdisciplinary Major of
Ocean Renewable Energy Engineering, National Korea Maritime & Ocean
University. He has over six years of working experience in management
information system include databases administrator, data warehouse, data
analytics BI, cloud computing, and API integrate multiple systems. He was
an Associate Software Engineer with the Department of International Finan- JUN-HO HUH (Member, IEEE) received the B.S.
cial Reconcile System, VNG Corporation, Vietnam, from June 2015 to degree in science from the Department of Major of
December 2016. He was a Senior Consultant with the BI Reporting and Applied Marine Sciences, the B.E. degree in engi-
Data Warehouse for Masan Group, Vietnam, from January 2017 to January neering (double major) from the Department of
2018. From January 2018 to June 2018, he was a Senior Consultant with Computer Engineering, Jeju National University at
BI Reporting and Data Warehouse for VP Bank & Techcombank (Hanoi Ara, Jeju, Republic of Korea, in August 2007, the
Head Quarter), Vietnam. He was a Senior Database Administrator and M.A. degree in education from the Department of
Data Warehouse Analysis for Dat Viet VAC Group Holding, Vietnam, from Computer Science Education, Pukyoug National
April 2018 to April 2019. He was a Senior Software Engineer with the University at Daeyeon, Busan, Republic of Korea,
Department of International Financial Reconcile System, VNG Corporation, in August 2012, and the Ph.D. degree in engi-
Vietnam, from April 2019 to March 2020. Since September 2020, he has neering from the Department of Major of Computer Engineering, Graduate
been a Research Assistant with the Big Data Center for Total Lifecycle of School, Pukyoug National University at Daeyeon, in February 2016.
Shipbuilding and Shipping, National Korea Maritime & Ocean University. He finished the Cooperative Marine Science and Engineering Program,
He received the Best Paper Award at the 15th International Conference on Texas A&M University at Galveston, USA, in August 2006. He was the
Multimedia Information Technology and Applications (MITA 2019). General/Head Professor of Catholic University, Pusan on International Game
Exhibition G-Star 2017 (G-Star 2017). He was a Research Professor with
Dankook University at Jukjeon, Yongin, Republic of Korea, from July
2016 to September 2016. Also, he was an Assistant Professor with the
Department of Software, Catholic University of Pusan, Republic of Korea,
HAN-JONG KO received the B.S. degree from from December 2016 to August 2019. Also, he was an Assistant Professor
Jeju National University at Ara, in 1999, and the with the Department of Data Informatics, National Korea Maritime & Ocean
M.S. and Ph.D. degrees in animal environmental University, Republic of Korea, from September 2019 to September 2021,
science from Seoul National University, in 2004. where he has been an Associate Professor (Tenured) with the Department of
He is a Full Professor (Tenured) with the Depart- Data Informatics/Data Science, since October 2021. Also, since September
ment of Agricultural Science, Korea National 2020, he has been the Center Chair (Director) of the Big Data Center for
Open University, Republic of Korea, where he Total Lifecycle of Shipbuilding and Shipping, National Korea Maritime &
has been since 2024. Also, he was an Assistant Ocean University, where he has been a Join Associate Professor (Tenured)
Professor (Tenure Track) with the Department of with the Global R&E Program for Interdisciplinary Technologies of Ocean
Agricultural Science, Korea National Open Renewable Energy (BK 21 Four Research Group) of Interdisciplinary Major

126516 VOLUME 12, 2024

H.-D. Thai et al.: Coffee Bean Defects Automatic Classification Realtime Application Adopting Deep Learning

of Ocean Renewable Energy Engineering, since September 2022. He is awarded the Commendation for Meritorious Service in the Promotion
the Book Author of Smart Grid Test Bed Using OPNET and Power Line of Busan’s Data Industry, in December 2023 (Commendation by the
Communication (pp.1–425, IGI Global, USA, 2017) and Principles, Poli- Mayor of Busan Metropolitan City, Korea Government). Also, he was
cies, and Applications of Kotlin Programming (pp.1–457, IGI Global, USA, the Organizing Chair of the 15th International Conference on Multime-
2023). He has authored/edited ten books and edited ten special issues in dia Information Technology and Applications (MITA 2019: University of
reputed Clarivate Analytics Index journals. Also, he has published more than Economics and Law (UEL), Vietnam National University, Ho Chi Minh
110 articles in Clarivate Analytics Index (SCI/SCIE/SSCI indexed) with over City, Vietnam). Also, he was the Organizing Chair of the 17th Interna-
3600 citations and has an H-index of 33. tional Conference on Multimedia Information Technology and Applications
Dr. Huh has been a Technical Committee (TC) Member of International (MITA 2021: Jeju KAL Hotel, ROK). He was a Managing Editor (ME)
Federation of Automatic Control (IFAC), CC 1 (Systems and Signals), and of Journal of Information Processing Systems (JIPS), Korea Informa-
TC 1.5. (Networked Systems), since 2017. Also, since 2017, he has been tion Processing Society (SCOPUS/ESCI indexed), from January 2020 to
a Technical Committee (TC) Member of IFAC, Computers, Cognition and December 2021. Also, he was a Managing Editor (ME) (January 2017–
Communication (CC 3), and TC 3.2 (Computational Intelligence in Control). December 2022) and is an Associate Editor (AE) of Journal of Multimedia
And, since 2017, he has been a Technical Committee (TC) Member of IFAC, Information System (JMIS), Korea Multimedia Society (EI/KCI Indexed).
CC 7 (Transportation and Vehicle Systems), and TC 7.2. (Marine Systems). Since July 2023, he has been the Editor-in-Chief (EiC) of Journal of
Also, since 2020, he has been a Technical Committee (TC) Member of Global Convergence Research, Global Convergence Research Academy,
IFAC, CC 2 (Design Methods), and TC 2.6. (Distributed Parameter Systems). Pukyoug National University at Daeyeon. Also, he is an Associate Edi-
He received Best Paper Minister Award (Ministry of Trade, Industry and tor (AE) of Journal of Information Processing Systems (JIPS), Korea
Energy, Korea Government) the 16th International Conference on Control, Information Processing Society (SCOPUS/ESCI indexed). Also, he is an
Automation and Systems, in October 2016, ICROS with IEEE Xplore. Associate Editor (AE) of Human-Centric Computing and Information
Also, he received Springer Nature Most Cited Paper Award, Human-Centric Sciences (HCIS) (SCIE IF=6.60). For more information visit the link
Computing and Information Sciences Most Cited Paper Award 2019 (rese- (https://scholar.google.co.kr/citations?user=cr5wjNYAAAAJ&hl).
arch published in the journal, from 2016 to 2018; SCIE IF = 6.60). He was

VOLUME 12, 2024 126517