Final year thesis presentation on

YOLOv5-Based Detection and Classification of Bangladeshi Local Drug: A

Framework with validation and drug Recommendations

Submitted By
Anik Deb Nath Fahamida Rahman Sahat Al Iqbal
ID: 1803510201673 ID: 1803510201698 ID: 1803510201677

In partial fulfillment of the degree of

Bachelor of Science in Computer Science & Engineering
Under the Supervision of
MS. SABRINA TARANNUM
Lecturer
Department of Computer Science & Engineering
Premier University, Chittagong
 TABLE OF CONTENTS

 Introduction  System Flow chart

 Key Objectives  Dataset Details

 Motivations  Model Overview & Training

 Tools Used  Results

 Related Works  Limitations

 Background Study  Future Works

 Key Contributions  Conclusions

 Methodology  References

Dept. of CSE, PUC

 INTRODUCTION

 Object detection, a vital aspect of our drug package recognition system, accurately
identifies medication packaging in images or videos.
 With object detection, we're making medication identification faster, more precise,
and simpler for everyone involved in the process.
 Our system quickly and accurately identifies medication packaging using cutting-
edge deep learning techniques like YOLOv5.

Here we introduce a YOLOv5-Based system for Detection and Classification of

Bangladeshi Local Drug: A Framework with validation and drug
Recommendations.
.
Dept. of CSE, PUC
 KEY OBJECTIVES
To Detect Bangladeshi local drug using yolov5 based on generic name and validation of detected result.
Also recommend more similar drugs based on detected generic name for right detection and to give
alarm sound for wrong detection.
Generic: Ciprofloxacin Incorrect
detection
Class: Antibiotics

Correct
detection
Antibiotics

(a) (b)

Figure: Example of sample drug image: (a) the drug image with generic (b)
detected drug image and (c) suggestion of detected generic drug

Dept. of CSE, PUC

(c)
 MOTIVATIONS

 Manual inspection techniques that are based on tradition take a lot of time and are

prone to human mistake

 Provide a solution for Bangladeshi local drugs, considering unique packaging

characteristics.

 Identify generic medicine names on packaging properly to protect public health

 Fill in the blanks in the current solutions by providing a thorough model complete with

alert mechanisms and validation recommendations.

Dept. of CSE, PUC

 TOOLS & LIBRARIES USED

 Google Colab  Matplotlib

 LebelImg  Pandas

 Photoshop  cv2 (OpenCV)

 Tesseract OCR  Pydub

 Pytesseract  Yaml

 IPython

Dept. of CSE, PUC

 RELATED WORKS
 Sheng-Luen Chung et al. (2020) contributed to the development of a drug identification model using deep
learning technologies, as documented in their paper titled "A drug identification model developed using deep
learning technologies: experience of a medical center in Taiwan.
Limitations: In dataset diversity, computational resources, image quality variance, interpretability, and
maintenance requirements.

 Youjin Kang, SangKeun Lee, Dong-Hwa Jeong, and Kang-Min Kim Junyeong Heo, "An Accurate Deep
Learning–Based System for Automatic Pill Identification: Model Development and Validation," Journal of
Medical Internet Research, no. 36637893, Jan 2023..
Limitations: The system only recognizes capital letters, excluding lowercase letters and symbols like "mg,“
due to data constraints. This limitation can be addressed by collecting more diverse data in future studies.

Dept. of CSE, PUC

 RELATED WORKS (Cont.)
 Yu-Sin You et al. (2023) proposed a Novel Two-Stage Induced Deep Learning System. They focused on
developing Deep learning classification systems for unpackaged pills or drugs with the same type of
packaging.
Limitations: Diverse packaging needs to be validated in real-world settings. Factors such as varying lighting
conditions, camera angles, and background clutter may influence the system's performance in practical
applications

 Hyuk-Ju Kwon et al. (2022) developed a deep Learning based system and Detection Technique with Least
Image-Capturing for Multiple Pill Dispensing Inspection using a method optimized for multi object detection
and recognition based on the YOLOv4 model.
Limitations: Potential challenges in generalizing to various pill dispensing scenarios, and potential trade-offs
between accuracy and image-capturing efficiency.

Dept. of CSE, PUC

 BACKGROUND STUDY

Detecting and classifying locally prevalent drugs in Bangladesh is critical for public safety. Traditional

methods are slow and costly, prompting the need for efficient technologies. Previous research in drug

detection has mainly focused on machine learning and computer vision. There's a lack of research

specifically targeting Bangladesh's local drugs. Validation methods for detection models are crucial but

underexplored, and the broader socio-economic and public health impacts of drug abuse need

consideration. Hence, this study proposes a YOLOv5-based framework for detecting Bangladeshi local

drugs, accompanied by recommendations for drug enforcement and public health interventions.

Dept. of CSE, PUC

 KEY CONTRIBUTIONS

 Dataset: Curated a dataset with medicine packaging images labeled with generic names, which

anyone can use in future for research.

 YOLO Integration: Used YOLOv5 for real-time generic name identification, enhancing

pharmaceutical validation.

 Verification: Integrated alert systems for inaccurate detections, improving system reliability.

 Recommendation: After successful detection our model can able to recommend similar drug

Dept. of CSE, PUC

 PROPOSED METHODOLOGY
Collect drug package images

Annotated Drug Dataset

Train

yolov5 model

Detect by trained weight

Detection

From txt file and detected image

Extract detection result

Match with csv dataset

Verification

Verification result
Give wrong alert
& recommendation
Dept. of CSE, PUC
 PROPOSED SYSTEM FLOW

Dept. of CSE, PUC

 DATASET DETAILS
 Source of images: From local drug store.
 Number of images: 2328
 Number of type use as class: 38

The name of the classes are used in this drug dataset are shown below:

Mental Health, Hypertension/Blood Pressure, Antibiotics, Gastrointestinal, Vitamins & Minerals, Diabetes,
Allergies, Cough & Cold, Inflammation, Pain-&-Fever-Reliever, Pain-Reliever, Nausea & Vomiting, Fungal
Infections, Cholesterol, Cardiovascular, Constipation, Asthma, Diarrheal, Muscle Relaxant, Antiseptic, Asthma &
Allergy, Asthma & COPD, Menstrual, Herbal Preparation, Parasites, Urological treatment, Neuropathic Pain,
Neurological/Brain Disease, Migraine, Nasal Decongestant, Multi-Purpose Medication, Cancer Treatment,
Thyroid Hormones, Birth Control Pills, Viral Flu, Antiplatelet Medications, Anemia, Arthritis.

Dept. of CSE, PUC

 HOW TO MAKE DATASET

To train our dataset in yolov5 model there are number of steps are involved for making dataset which
shows in figure below:

Figure: Flow diagram of making dataset

Dept. of CSE, PUC

 HOW TO MAKE DATASET (Cont.)

 Data Preprocessing:
After collection of drug images need to preprocess. Example of preprocess which we used: figure (a)
is captured drug package image and figure (b) is preprocessed drug package image.

Figure:(a) Figure: (b)

Dept. of CSE, PUC

 HOW TO MAKE DATASET (Cont.)
 Data augmentation:
Some images we need to use augmentation due to lack of data. Example of augmentation which we
used: figure (a) main image and figure (b) augmented images.

Figure:(a)

Dept. of CSE, PUC

Figure:(b)
 HOW TO MAKE DATASET (Cont.)
 Data annotation: In figure (a) Flow diagram of image annotation is
shown:

Figure:(a)

Dept. of CSE, PUC

 YOLO MODEL
 You Only Look Once (YOLO) is a state-of-the-art, real-time object detection algorithm introduced in 2015 by
Joseph Redmon. Some of the reasons why YOLO is leading the competition include its:
 Speed
 Detection accuracy
 Good generalization
 Open-source

 One of the most popular version is YOLOv5 which Released by Glenn Jocher in June 2020. The release
includes four different model sizes: YOLOv5s (smallest), YOLOv5m, YOLOv5l, and YOLOv5x (largest). A few
examples of use cases of YOLOV5 are:
 Face Mask Detection,
 Object Recognition,
 Speed calculator,
 Vehicle Tracker, and so on
Dept. of CSE, PUC
 HOW YOLO WORKS
1. Grid Division: The input image is divided into a grid of cells.
2. Prediction per Grid Cell: Each grid cell predicts bounding boxes
and class probabilities for objects within that cell.
3. Bounding Box Prediction: For each grid cell, YOLO predicts
bounding boxes.
4. Bounding boxes consist of attributes: (x, y) coordinates of the
center, width, height, and a confidence score indicating the
probability of containing an object.
5. Class Prediction: YOLO predicts the probability distribution over
all classes for each bounding box.
6. Post-processing: Non-maximum suppression (NMS) is applied to
remove duplicate and low-confidence bounding boxes. Only the Figure: Object detection stages
most confident bounding boxes for each object are retained.
7. Output: The final output includes the retained bounding boxes
Dept.
along of their
with CSE, corresponding
PUC class labels and confidence scores.
 YOLOv5 TRAINING PROCEDURES

 Split dataset:
It is also important to know the directory structure of
the dataset before using it for training.

 Configure YAML file:

This file contains the path to the training and
validation data, along with the class names. Figure:(a)

 Clone / Download the YOLOv5

From https://github.com/ultralytics/yolov5.git

 Pre-trained weight :
(Yolov5s.pt,Yolov5m.pt, Yolov5l.pt, Yolov5x.pt)
Dept. of CSE, PUC
Figure:(b)
 MODEL TRAINING WITH (YOLOv5x)
 Command to train yolov5 model:

!python train.py --data data/custom.yaml --weights yolov5x.pt --img 640 --epochs 100 --batch-size 16

 --data: Path to the dataset YAML file.

 --weights: Model weights file for training.

 --img: Image size for resizing (640 pixels).

 --epochs: Number of epochs for training.

 --batch-size: Number of samples loaded into one batch during training.

Dept. of CSE, PUC

 TRAINING RESULT (YOLOv5x)
The YOLOv5x model was able to achieve an mAP of 72.8% at 0.5 IoU and 48.3% at 0.5:0.95 IoU. Summary is here:

Dept. of CSE, PUC

 TRAINING RESULT (YOLOv5x)
All the curve of Loss and mAP results after training the YOLOv5x model for 100 epochs.

Dept. of CSE, PUC Figure : Loss and mAP results

 VALIDATION RESULT

Dept. of CSE, PUC

Figure: Validation result

 INTERFACE RESULT
Command for detection:

!python detect.py --weights runs/train/exp/weights/best.pt --img

640 --conf 0.1 –source /content/drive/MyDrive/yolov5/testImages
--save-txt --save-conf

Most important part here is the detect.py

--weights: trained weights (best.pt) to be used for inference.
--source: the path where the images or videos are present. Figure:(a)
--save-txt: save the bounding box information
--save-conf: save the confidence score

The following figure shows the image inference results. Figure

(a) shows input image and (b) shows its result.

Dept. of CSE, PUC

Figure:(b)
 INTERFACE RESULT
There are so many interface result shown in below:

Dept. of CSE, PUC

 TEST CASE – 1 (VARIFICATION)

In Figure (a) shows the result drug type and confidence rate . Now
we extract the detected result for the image.

EXTRACTION RESULT:

Detected class name : Pain-&-fever-reliever

Generic name: Paracetamol

VERIFICATIOIN: (a)

For verification we use another tabular dataset which is shows in

figure(b) below: Here we see that model Predict correctly.

Dept. of CSE, PUC

(b)
 TEST CASE – 1 (RECOMMENDATION)
The identification results are correct. As correct prediction model can give suggestion some drug according to match
generic name. Suggestion result shown below in figure:

Dept. of CSE, PUC

Figure: Recommendation result for generic ‘Paracetamol’
 TEST CASE – 2 (VARIFICATION)
In Figure (a) shows the result drug type and confidence rate . Now
we extract the detected result for the image.

EXTRACTION RESULT:

Detected class name : Mental Health

Generic name: Atorvastatin

(a)
VERIFICATIOIN:

For verification we use another tabular dataset which is shows in figure(b) below: Here we see that model
Predict Incorrect. So wrong alarm sound is occurred as well as no recommendation.

Dept. of CSE, PUC

(b)
 CONCLUSION

In conclusion, we can say, the system provides matching results and recommendations based on
the identified drug's generic name. Experimental results demonstrate the model's efficacy in
detecting and classifying drugs, generating warnings for incorrect dispensing packages, and
offering accurate recommendations for correct detections, with an impressive average precision
of 72.8% at IoU 0.5 (mAP50) and 48.3% across IoU thresholds ranging from 0.5 to 0.95 (mAP50-
95). However, these results highlight the efficiency and effectiveness of the developed YOLOv5
model in drug identification, offering a practical solution for familiarizing users with generic drug
names and promoting accessibility in the pharmaceutical domain.

Dept. of CSE, PUC

 LIMITATIONS

 Data availability, especially for rare drugs, hampers model generalization.

 Dependence on textual information may limit effectiveness in non-language-centric

scenarios.

 Dataset inconsistency during training due to variations in real-world medication packaging.

 Challenges in maintaining dataset consistency stem from differences in picture quality.

Dept. of CSE, PUC

 FUTURE WORKS

 Increasing dataset size to encompass diverse medicine packages for improved model
resilience.
 Transitioning from computer-based simulation to deep learning embedded systems for real-
world testing.
 Potential development of a complete robotic system for drug detection, identification, and
confirmation.
 Utilizing AI-enabled mechanisms to create precise, safe, and accurate medication environments
in healthcare.
 By following this process further add also Bangla language named drug to enhance the
detection of drug.

Dept. of CSE, PUC

 REFERENCES
 Chirag Gupta,Ayush Rawat Ayushya Verma, "MedsGen: A Comprehensive Platform forGeneric Medicine
Identification and Acces," International Journal of Engineering Research & Technology (IJERT), vol. xii, no. 05,
pp. 01-26, May 2023.
 J., Divvala, S., Girshick, R. & Farhadi Redmon, "You only look once: Unified, real-time object detection," vol.
779–788, June 2016.
 J. & Farhadi Redmon, "Yolo9000: Better, faster, stronger," In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, vol. 7263–7271, no. 2017, 2017.
 A., Wang, C.-Y. & Liao, H.-Y. M. Bochkovskiy, "Yolov4: Optimal speed and accuracy of object detection," arXiv
preprint , 2020.
 Guen-Han Li, Po-Cheng Su,Roy Chaoming Hsu Hui-Chuan Lin, "A DEEP LEARNING-BASED ACCURATE
DRUG DETECTION, IDENTIFICATION AND CONFIRMATION MECHANISM FOR MEDICATION
DISPENSING PACKAGE OF INPATIENTS," International Journal of Computer Science Engineering and
Applications, vol. 12, p. 15, April 2022.

Dept. of CSE, PUC

 REFERENCES (Cont.)
 Sheng-Luen Chung, Chih-Fang Chen Hsien-Wei Ting, "A drug identification model developed using deep
learning technologies: experience of a medical center in Taiwan," IEEE , vol. 20, April 2020.
 Youjin Kang, SangKeun Lee, Dong-Hwa Jeong, and Kang-Min Kim Junyeong Heo, "An Accurate Deep
Learning–Based System for Automatic Pill Identification: Model Development and Validation," Journal of
Medical Internet Research, no. 36637893, Jan 2023.
 Adewumi E, Amawi RM Al-Hussaeni K Karamitsos I, "CNN-Based Pill Image Recognition for Retrieval
Systems," Applied Sciences, vol. 8, p. 13, April 2023 Michael Williams, "Automated Drug Recognition System
for Pharmacy Automation," journal of Pharmaceutical Science and Technology, 2018.
 D. Davies, "Collect and Label Images to Train a YOLOv5 Object Detection Model in PyTorch,"
https://wandb.ai/onlineinference/YOLO/reports/Collect-and-Label-Images-toTrain-a-YOLOv5-Object-Detection-
Model-in-PyTorch--VmlldzoxMzQxODc3, p. 1, December 2021.
 "YOLO v5 model architecture [Explained]," OpenGenus IQ: Computing Expertise &Legacy.
https://iq.opengenus.org/yolov5/, August 2022

Dept. of CSE, PUC

 REFERENCES (Cont.)
 F. Sun, J. Gu, and L. Deng H. Liu, "SF-YOLOv5: A Lightweight Small Object Detection Algorithm Based on
Improved Feature Fusion Model," https://doi.org/10.3390/s22155817., vol. 22, p. 5817, August 2022.
 T.-Y. Lin et al., "Microsoft COCO: Common Objects in Context," Springer International Publishing, vol. 8693,
pp. 740–755, 2014.
 V.-H. Nguyen, and J.-H. Huh T.-N. Pham, "Integration of improved YOLOv5 for face mask detector and auto-
labeling to generate dataset for fighting against COVID-19," The Journal of Supercomputing, no. May 2023, p.
79, January 2023.
 Karamitsos I, Adewumi E, Amawi RM Al-Hussaeni K, "CNN-Based Pill Image Recognition for Retrieval
Systems.," Applied Sciences., vol. 8, p. 13, April 2023.

Dept. of CSE, PUC

THANK YOU
Feel free