See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/329056573

Object Sorting using Image Processing

Conference Paper · May 2018

DOI: 10.1109/RTEICT42901.2018.9012110

CITATIONS READS

13 11,700

3 authors, including:

Rahul Soans Pradyumna G R

Ritsumeikan University NMAM Institute of Technology
8 PUBLICATIONS 34 CITATIONS 16 PUBLICATIONS 41 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Rahul Soans on 20 November 2018.

The user has requested enhancement of the downloaded file.

 2018 3rd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT-2018), MAY 18th & 19th 2018

Object Sorting using Image Processing

Rahul Vijay Soans Pradyumna G.R. Yohei Fukumizu
Department of Electronics and Department of Electronics and Department of Electronics and Electrical
Communication Engineering Communication Engineering Engineering
N.M.A.M. Institute of Technology N.M.A.M. Institute of Technology GSSE, Ritsumeikan University
Karnataka, India Karnataka, India Shiga, Japan
[email protected] [email protected] [email protected]

Abstract— Automation has led to the growth of industries in and servo motors. Raspberry Pi is a portable computer with the
recent years. For better performance of industrial process ARM11 architecture which runs on Linux Debian environment.
automated machines are used. Image processing has led to The proposed system has a conveyor belt on which the objects
advancements in applications of robotics and embedded systems. to be separated are placed. A camera constantly captures the
Sorting of objects are usually done by humans which takes a lot objects passing on the conveyor belt. The captured frames are
of time and effort. Using Computer Vision techniques, a conveyor processed to identify and recognize various objects and then the
belt system is developed using stepper, servo motors and decision is made for classifying the products. Stepper motors
mechanical structures, which can identify and sort various and servo motors are used to sort the objects running through
objects. This reduces human effort, time consumed, and also
the conveyor belt by mechanical structures which directs the
improves the time to market the products.
objects to their destination. Electrical actuators will be
Keywords— Conveyor Belt, SIFT, Raspberry Pi 3, OpenCV activated which will push away the objects into a different path
in a conveyor belt or into a container.
I. INTRODUCTION Object sorting machine has been developed previously
using optical sensors and color sensors [4]. Also using Image
Application of image processing in industries has gained a
processing techniques, the colored objects are detected and
lot of scope. The research on automation and robotics has
segregated using a robotic arm [5]. Object tracking robots are
shown importance in industries, defence, surveillance and
also developed using adaptive color thresholding technique [6].
security using image processing [1] [2]. Object sorting is one of
There are various recognition techniques like boosted learning
the challenges faced by the industries. The main objective is to
algorithms, Bag of Words model, template matching, blob
build a conveyor belt system that is capable of identifying and
analysis, segmentation. Recognition algorithms are used to
sorting various objects by using image processing.
identify the object of interest in video or real-time web camera.
The system should detect the objects based on their The approaches based on a Bag of Words or Bag of features
properties. Objects will have different shapes and colors. This has many applications like object and scene recognition [7] [8].
process is made easy by using automation. Automation
The Bag of features or Bag of words is a well-known
provides mechanical assistance by using a control system. This
classification method for object recognition. An image feature
reduces manual efforts done by a human, time consumed and
identification algorithm has key point detection and descriptor
also improves the time to market. The aim of the proposed
extraction using SIFT method [9] because of its invariance to
project is to develop a system that can identify various objects
the scale, the orientation and almost to the illumination. Key
running through a conveyor belt and sort them based on their
feature points are extracted from an image and based on its
properties by designing mechanical structures, hence
feature, a descriptor vector is created. Bag of Words is a
automating the process of segregation of products.
representation based on visual histogram. Features are created
Object Recognition is the most important task in the by clustering all the extracted features from training images.
computer vision field. The increasing demand for real time Each feature is mapped into visual word and it is represented
image processing, has led to a great deal of research in Object by the histogram of visual words. Later, Support Vector
Recognition algorithms [3]. Object Recognition is widely used Machine (SVM) is used as supervised learning classifier while
in various applications like automation, surveillance, medical testing an unknown image.
field, etc. The goal of object recognition is to automatically
detect the objects in the screen and classify them according to II. SYSTEM DESIGN
their properties. This process has to be repeated for all the
frames of the captured images. The region of interest is This design uses a low cost hardware and open source
determined by training a model from object samples. software for achieving the goal. Raspberry Pi 3 and a computer
are used as the controlling unit. The proposed system has a
Using Image processing techniques, the various objects or conveyor belt which is driven by stepper motors and pulley
products are identified. OpenCV (Open Source Vision Library) arrangement. Object is fed from one end of the belt. The pulley
is used for image processing using with Python scripting that drives the conveyor belt is called drive pulley and the end
language. OpenCV has powerful image processing functions pulley is called as the idler pulley. USB camera is used to
which is suitable for real time. Raspberry Pi is used as the continuously monitor the objects and identify them. The
controller to control hardware components like stepper motors

978-1-5386-2440-1/18/$31.00 ©2018 IEEE

recognized objects are sorted by the actuators which can be a B. Object Recognition using Bag of Features
linear actuator or a servo motor. Bag of Visual Words or Bag of Features is a popular visual
data classification method. Image classification needs
information from the images. It extracts the features of the
images and forms a codebook and is used to compare it with
features of unknown images. The various steps involved in Bag
of Features method is shown in Fig. 3.

Fig. 1. Basic block diagram of object sorting system

The basic block diagram of the proposed system is shown

in Fig. 1. This design uses a low cost hardware and open source
programming language and image processing libraries for
achieving the goal of sorting objects. CPU is used to process
the images from the webcam and Raspberry Pi 3 [10] with
Linux operating system is used as the controlling unit to drive
various hardware devices. The CPU acts as a server and the
Raspberry Pi acts as a client. The CPU will send a command to
the Raspberry Pi when an object a particular object is detected.
The Raspberry Pi client will receive the command and runs the
Fig. 3. Steps involved in Bag of Features method
corresponding script for driving the stepper motor, based on the
object detected. The server and client model separates the The database consists of N classes to be trained for
software and hardware operation and will speed up the process classification. The first step is the feature extraction or keypoint
in real time. extraction. Meaningful blob like feature points on the object
can be extracted by using SIFT method. These are extracted
A. Work Flow from training image sets, and stored as numpy array. Vector
The flow chart of proposed system is shown in Fig. 2. The quantization technique is used to cluster data points in the
proposed system has a conveyor belt which runs with the help feature space. Each cluster is a “vocabulary" which represents
of stepper motors and corresponding pulleys at the motor and the pattern that exist in the key points. Then these clusters are
its opposite ends which constantly run at a desirable speed. The represented as the histogram of occurrences to define a Bag of
stepper motors are initialized to run the conveyor belt. Material Words containing meaningful visual words. Finally the
is fed on the feed-side of the belt and landed on the rotating classifier stage classifies the vocabulary based on the prediction
conveyor belt, and they rely on the conveyor belt friction to be on the clusters of feature space.
delivered to discharge end. USB camera is used to continuously
monitor the objects and identify them. Once the objects are 1) Feature Extraction and Description: Key points are
recognized or classified into a particular group, the actuators important features that is widely utilized in the image analysis.
are activated thereby sorting the objects. Local feature extraction is based on local geometrical features
such as Scale Invariant Feature Transform (SIFT) which is the
successful algorithm for feature detection. To perform a good
recognition, the features extracted from the training image must
be recognised even under various image scale, noise and
illumination. Important points usually exist in object edges and
corners. A good feature should be invariant to rotation and
illumination. A point feature can be edge or a corner. SIFT
extracts blob like feature points which are large binary objects.
Which is then stored as a descriptor with is invariant to scale,
illumination, and rotation.
SIFT feature is extension of Harris corner detection. Harris
corner detection extract same features even if the image is
rotated since they find corners. But it is not scale invariant.
This is because of the same sized window cannot detect the
same features if the image is scaled into different size. In order
to detect the scaled images, larger window is required. Scale
space filtering is used which can detect blob features with
different σ values. It is a scaling parameter from which various
sizes of blobs can be detected. Hence, local maxima is found at
(x,y,σ), where x and y are the spacial coordinates. Also the
Fig. 2. Flow chart of object sorting system
maxima is detected in different scales by comparing the pixels which are not of the same object but within the same class of a
with neighbouring pixels and pixels from previous and next particular object. K-means clustering is used for quantizing the
scale as shown in Fig. 4. If there is local extrema, then it can be feature points in feature space before providing the data to the
considered as a potential keypoint. SVM classifier.

K-means clustering is a vector quantization technique in

signal processing. K-means clustering will partition ‘n’
observations into ‘K’ clusters where each point is assigned to
the cluster with the nearest mean. The K-means clustering
algorithm is done by selecting initial centroids at random at
the beginning and assign each key point to the cluster with the
nearest centroid. For the next iteration, a new centroid is
calculated which is the mean of new cluster points. This
Fig. 4 Scale-space extrema detection process is repeated until the centroid won’t change further.
Also contrast thresholding is performed on the intensity to While implementing in Python using OpenCV function, the
improve the accuracy of the location of extrema. SIFT feature value of ‘K’ is controlled depending on the number of clusters
uses orientation assignment to achieve scale invariance. required.
Gradiant magnitude and direction of the keypoint 3) Histogram Representation: K means provides labels for
neighbourhood are found as shown in Fig. 5. Also orientation each data points which can be then given to SVM classifier.
histogram with 36 bins is created which provides keypoints in Data points are represented as a vector which contains the
different directions. Then, the keypoint descriptors are created weighted count of each visual word. Each of the clusters’
by considering 16x16 neighbourhood around the keypoint and center is considered to represent a word by quantization. The
sub divided into 4x4 blocks as shown in Fig. 6. 8 bin number of occurrences of each word is calculated which
orientation histogram is created for each sub block, providing indicates the probability of words in each class of objects.
128 bin values represented as a vector. Fig. 7 depicts feature Subsequently, it is a step towards computing a codebook or a
extracted from various objects from the database. dictionary of several classes of object.

4) SVM Classification: While testing a new set of images,

classification method is used to predict which class a new data
point will be in. SVM is a linear classifier which builds a
model that assigns new examples to one of the classes, from
the training sets. An SVM model predicts and maps new
points with clear separation gap. Data points are dimensional
vector which is given to the SVM classifier to train a model.
Fig. 5 Gradient magnitude and direction There are many hyperplanes that separate the two different
classes. Best hyperplane with maximum separation is
considered which is also called as ‘Optimal Hyperplane’.

C. Conveyor Belt Construction

A Conveyor belt is the carrying medium for the products
while packaging. It consists of two or more pulleys, with a loop
of carrying medium that rotates around them. Pulleys are
powered which causes the belt to rotate. The main pulley is
Fig. 6 SIFT orientation histogram
called as drive pulley and the dummy pulley is called the idler
pulley. The basic pulley system consists of a driver pulley and
driven pulley as shown in Fig. 8. A rubber belt is used to link
the two rotating shafts. The ideal pulley ratio between the
driver and the driven pulley is 2:1.

Fig. 7 SIFT feature points extracted from the objects

2) Visual Vocabulary: After the extraction of descriptors, the

next step is to cluster descriptors using vector quantization Fig. 8. Basic belt driving mechanism
technique. A model is constructed that represents all the images
 The stepper motor drives the pulley of the belt and causes Conveyor spacers are used to adjust the width of the belt.
the conveyor to rotate. A gearing mechanism is used with the Also the covers are used to cover the pulley mechanism. The
motor, to drive enough load on the conveyor. The gearbox is cross sectional view and 3D view of conveyor spacers and
used with the induction motor to drive the pulley. Various covers are shown in Fig. 15 and Fig. 16.
mechanical parts like aluminium frame, chassis, cylindrical
pulley, pulley holders, belt cover, rubber belt, conveyor spacers
and covers are used to construct the prototype. The gear box
shaft is connected to the drive pulley. The cross-sectional and
3D view of cylindrical pulley and holder are shown in Fig. 9
and Fig. 10. Fig. 15 Conveyor spacers

Fig. 9 Cylindrical pulley

Fig. 16 Conveyor covers

The fully assembled Conveyor belt model is shown in Fig.

17. The conveyor belt setup has the following specifications:
Fig. 10 Pulley holder
 Belt Width: 150 mm
Once the pulley systems are built, it is then attached to the  Belt Length: 2000 mm
aluminium frame where the idler pulley acts as the support  Drive Roller Diameter: 30 mm
pulley for the free motion of the belt. The aluminium frame and  Max Load: 2 Kg
chassis acts as the strong base for the conveyor belt. The frame
width is two meters long which acts as a support wall for the
belt. The gear box shaft is attached to the cylindrical pulley and
screwed tightly. Pulley holders are also used at the ends to hold
the pulley and provide free circular motion for the pulley. The
cross-sectional view and 3D view of aluminium frame and
chassis are shown in Fig. 11 and Fig. 12.

Fig. 17 Conveyor belt model [11]

Fig. 11 Conveyor aluminium frame

III. RESULTS AND DISCUSSIONS
Object recognition is done in real time using a Pi camera by
training various images. It uses SIFT extraction and SVM
classification for object recognition. The training process is
Fig. 12 Conveyor aluminium chassis implemented using Python 2.7 and OpenCV library in
Raspberry Pi 3 model with quad core processor, 1 GB RAM,
Once the pulley is attached to the motor using gears, the CPU at 1.20 GHz. The frame rate was found to be less than 10
belt cover is placed on the chassis. Later, the flat bed belt is fps at low resolution of 320*240. The training process produces
attached to the pulley on the cover. The tension of the belt can a classifier which is later used for testing. Also the conveyor
be adjusted based on the requirement. The cross sectional view belt is tested by driving the stepper motors.
and 3D view of conveyor belt cover and flat bed belt are shown
in Fig. 13 and Fig. 14. Later, the real time recognition is run on a better processor
with i7 processor, 8 GB RAM, CPU at 2.20 GHz. Commands
are sent to the Raspberry Pi using server and client model as
the objects are detected using USB camera. The client process
will be running continuously on the Raspberry Pi. Whenever a
Fig. 13 Conveyor belt cover particular set of objects are detected in the server computer, a
code is sent to the client python script running in Raspberry Pi
using socket communication. The client runs the corresponding
thread and activates the actuator. The frame rate in the
computer was found to be around 10 fps even at a higher
Fig. 14 Flat bed rubber belt resolution of 800*600. Thus, the real time recognition is
improved using this approach.
 Each class of object has around 10 images captured at the recognition depends on the lighting conditions and also the
various angles, sides and lighting conditions. In this performance of the computer. After object recognition, the
implementation, 3 different types of objects like boxes, cans delays between the actuators are made sure that it works
and bottles are used. Each individual object is trained with 10 synchronously for proper sorting of objects. In order to
images each. The accuracy can be improved by increasing the improve the accuracy, environment with constant light and
training samples which takes more time for training. The clear background must be used.
trained model is later used while testing of unknown image.
Fig. 18 shows the object recognition in real time using a ACKNOWLEDGMENT
webcam.
There have been several individuals who have played a role
in the development of this project. I express my deepest
gratitude and respect to my guide Mr. Pradyumna G. R.,
Assistant Professor, for his valuable guidance and
encouragement while doing the work.
Also special thanks and my sincere gratitude goes to my
guide Dr. Yohei Fukumizu, for his valuable suggestions and
providing me with all necessary laboratory facilities and
equipment required for this project.
I would also like to express my appreciation and thanks to
Fig. 18 Real time object recognition my family and friends who have assisted me throughout my
work.
The stepper motor is used with the various gear
combination to drive the belt on the conveyor setup. The
direction and speed of the conveyer belt can also be controlled. REFERENCES
The conveyor belt will be running continuously. The final
developed prototype is shown in Fig. 19. The stepper motors [1] J. Kramer and M. Scheutz, “Development environments for autonomous
are controlled by the Raspberry Pi 3 with motor drivers using mobile robots: A survey,” Autonomous Robots, vol. 22, pp. 101–132,
February 2017.
Python coding. The speed and the direction of the conveyor
[2] Luo, R.C. Chou, Y.T. Liao, C.T. Lai, C.C. Tsai, A.C. ,Nat. Chung Cheng
belt can also be controlled. The actuators are constructed by Univ., Min-Hsiung, “NCCU Security Warrior: An Intelligent Security
stepper motors. The speed of the conveyor belt is maintained, Robot System,” IEEE IECON, p2960, November 2007.
so that the more number of object frames can be acquired by [3] Jacinto C. Nascimento and Jorge S. Marques, “Performance Evaluation
the camera. of Object Detection Algorithms for Video Surveillance,” IEEE
Transactions on Multimedia, vol. 8, 2006.
[4] Timothy Henry, Laurence, Ishak, Ferry Jie, “Design and construction of
color sensor based optical sorting machine,” International Conference on
Instrumentation, Control and Automation (ICA), 2017.
[5] Viren Pereira, Vandyk Amsdem Fernandes, Junieta Sequeira, “Low cost
object sorting robotic arm using Raspberry Pi,” Global Humanitarian
Technology Conference - South Asia Satellite (GHTC-SAS), 2014.
[6] Rahul Vijay Soans, Ranjith, Abhilash Hegde, Chandra Singh and
Adithya Kumar, “Object Tracking Robot using Adaptive Color
Thresholding,” International Conference on Electronics and
Communication Systems (ICCES), 2017.
[7] A. Bolovinou, I. Pratikakis, and S. Perantonis, “Bag of spatio-visual
words for context inference in scene classification,” Pattern Recognition,
Fig. 19 Conveyor belt with actuators vol. 46, pp.1039-1053, 2013.
[8] T. Li, T. Mei, I.-S. Kweon, and X.-S. Hua, “Contextual bag-of-words for
visual categorization,” IEEE Transactions on Circuits and Systems for
Video Technology, vol. 21, 2011.
IV. CONCLUSION [9] Yuki Sakai, Tetsuya Oda, Makoto Ikeda, Leonard Barolli, “An Object
Tracking System Based on SIFT and SURF Feature Extraction
A visual system in real time is developed to recognize Methods”, International Conference on Network-Based Information
various objects in the scene using SIFT feature extraction and Systems, 2015.
SVM classification. The object classification was trained and [10] https://www.raspberrypi.org/products/raspberry-pi-3-model-b/
tested for the images taken by the USB camera. The real-time [11] https://jp.misumi-ec.com/
processing is done in a CPU with i7 processor, 8 GB RAM,
CPU at 2.20 GHz. Also the essential components of object
sorting system like conveyor belt and actuators are tested with
Raspberry Pi using Python scripting language. The accuracy of

View publication stats