2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT)

Bangalore, India. Oct 7-9, 2022

Virtual Canvas for Interactive Learning using

OpenCV
Palak Rai Reeya Gupta Vinicia Dsouza
Computer Engineering Computer Engineering Computer Engineering
Don Bosco Institute of Technology Don Bosco Institute of Technology Don Bosco Institute of Technology
Mumbai, India Mumbai, India Mumbai, India
[email protected] [email protected] [email protected]

Dipti Jadhav
Computer Engineering
Don Bosco Institute of Technology
Mumbai, India
[email protected]
2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT) | 978-1-6654-6855-8/22/$31.00 ©2022 IEEE | DOI: 10.1109/GCAT55367.2022.9971903

Abstract— The purpose of this study is to discuss the usage

of computer vision in educational applications. In recent years,
air writing has become one of the most challenging and After the pandemic caused in the year 2020 online mode
exciting research areas in image processing and pattern of education (e-learning) has become one of the most
recognition. The project employs object tracking techniques to important means of teaching and learning and hence one of
construct a motion-to-text converter that might be used as the applications of Virtual Canvas is making online lectures
software in the field of education to allow students and more fascinating and fun by acting as a white board on
teachers to write in the air. The project generates letters on the which lecturers can write or draw in real time with negligible
screen by utilizing computer vision to trace the path of a finger delay time and observe their gesture in the canvas (live
or an object. The resulting text can be utilized for a variety of window) with various user-friendly features making online
things, including solving graph-related challenges. Questions lectures more fun and interactive. Another application of
on logical reasoning (Mathematics), to write formulas or Virtual Canvas in education industry can be a need of
execute derivations, Kindergarten kids are being taught the dustless classroom during offline lecture, since chalk is made
alphabet, to replace chalk and board (dustless classroom) in of calcium carbonate and calcium sulfate which has an
offline classes, etc. For the deaf, it will be a strong way of impact on human health the user can face various health
communication. issue like allergies, itchy or watery eyes, respiratory problem
and many more. Also, use of plastic based marker and
Keywords—Air writing, gesture, object tracking, color
detection, image processing, Computer Vision.
boards, plastic made pen has an adverse effect on
environment which in return increases global warming which
I. INTRODUCTION can have a serious impact on not only human but on all the
living organism. And thus, Virtual Canvas can contribute to
The conventional art of writing is being supplanted by the chain of reduce, reuse and recycle. Hence, there are
digital art in the digital age. Many different methods of numerous scenarios in which the system can be quite useful.
writing are used in digital art, such as using a keyboard, a For example, in the sphere of education, long PowerPoint
touch-screen surface, a digital pen, a stylus, or electronic presentations can be replaced with a virtual canvas to
hand gloves. However, in this system, we utilize object increase interactivity between students and to incorporate
tracking using color detection techniques with the help of enjoyability within students. Using AI based Virtual Canvas
Python programming language and its various modules like can increase curiosity among students to develop and can
OpenCV, NumPy, tkinter, keyboard to produce a natural encourage innovation among them. This method has the
man-machine interface. With the growth of technology, the potential to be extremely useful in online as well as offline
demand for natural "human-computer interaction" solutions conversations.
to replace traditional systems is fast growing. So, introducing
artificial intelligence-based software in our education system II. BACKGROUND
which tracks the gesture of the user in real time and displays
the output with negligible delay time will make an impact on A. Literature Review
the environment by reducing the use of plastic based markers Around 10 literature articles were reviewed to obtain
replacing dust-filled classroom to dustless classroom and background knowledge on the vital concepts of detection and
making traditional teaching style more fun and interactive. masking technology and to explore what has been done
In this work, we propose a simple, enjoyable, and helpful already in these respective areas. To track the real time
system that allows users to write on a screen just by waving moving object in videos OpenCV is used along with
their colored finger or a colored object in the air. While the Convolution Neural Network and SVM classifier [2]. Using
different deep learning methods for object detection face
idea of waving fingers in the air to draw on a screen without
detection and pedestrian detection was discussed in [1]. The
touching it may sound odd at first, the idea is quite feasible
conclusion was made by modifying RCNN for object
when computer vision and Python are combined with object
tracking which gave best results. Image processing
tracking using color detection techniques.

978-1-6654-6855-8/22/$31.00 ©2022 IEEE 1

Authorized licensed use limited to: HKBK College of Engineering. Downloaded on December 06,2024 at 07:47:29 UTC from IEEE Xplore. Restrictions apply.
technique, that is to determine the shape, color and contour of chalk or colored markers which are considered harmful
of the object [4] for easy object tracking so that when two for the environment as well as human health.
shapes overlap the contour of the biggest shape is selected.
To enhance this color selection, segmenting the colors into D. Requirements
different parts and then filtering them will help in better x The model is created using the Python programming
detection [8]. After processing, background subtraction is language.
required to separate the foreground from background. For
this Mixture of Gaussian method was discussed in [3] for x The gestures will be controlled using OpenCV
detecting moving objects in outdoor environment. Gestures (Open Source Computer Vision) which makes use of
are the actions that the user makes that will be captured. A a video camera (laptop or computer) to track the user
brief about gestures is discussed in [5]. To recognize the gestures.
gestures, papers [6] and [7] gives brief review of methods x NumPy is used for signal processing and also for
that can be used. Making use of hardware device such as some numerical operations.
Raspberry Pi for object tracking and gesture recognition is
discussed in [9]. This makes use of the raspberry pi keys for x Tkinter is used to create the user interface at the start
features to change brush color and size. The applications of of the software and to select the object color.
Air Writing to solve real world problems using RCNN and
x The keyboard will be used to track the user inputs or
YOLO is proposed in [10].
shortcuts to save, undo, change color and clear the
B. Survey screen.
A survey was conducted to know what all challenges
were faced by teachers and students during online teaching
and learning period. About 8 out of 10 teachers said they III. PROPOSED SYSTEM
faced issues where they could not teach students in The project uses the Open source Computer Vision
traditional method that is using a white or black board. library to capture gestures as well as to do Image
Because of this teacher had to adapt to teaching using Segmentation and Image Processing [5]. The models
devices such as stylus or notepad where they could solve flowchart from start to end can be seen in Fig.1.
mathematical equations or formulae. But the main problem
here was that these tools were not provided by the schools or
colleges as some were not financially funded well. So,
teaching using shared images, PDFs or presentations came
in. This created lack of interactivity between students and the
motivation of learning within students decreased as there was
no fun way incorporated. Also, if the student had to explain
the idea or solve a numerical, they would need to solve it in
their books and then scan their page and send it to their
teacher or upload it on school websites. This resulted in
wastage of time and increased copying tendencies as the
teacher didn’t know who actually solved the numerical.
These were some of the problems faced by school and
college teachers.
When we consider preprimary school teachers, they had
issues related in teaching alphabets or numbers to students,
as gestures play a very important role here. Following the
gestures is how a preprimary school teacher chooses to teach
an alphabet.
Another survey was conducted for the students about
their overall learning experience in online mode and there
was a 50-50 positive-negative response. But the common
issue was the increased number of assignments because
teachers could not make the students solve them during
lecture slots.
C. Aim and Objective
The aim of our project is to make an inexpensive product
that lets the user write in air using any object with ease and
to improve the online learning experience by solving
problems that need the use of expensive products such as
stylus or touch screen.
The objective following this aim is to make Virtual
Canvas user friendly by incorporating features to save, undo,
maximize/minimize brush size and whiteboard for cluttered
backgrounds. Another objective is to create an Fig. 1. System Flowchart
environmentally friendly product which decreases the use

2
Authorized licensed use limited to: HKBK College of Engineering. Downloaded on December 06,2024 at 07:47:29 UTC from IEEE Xplore. Restrictions apply.
 Describing the flowchart, the first step for the user is to have one particular colored object available with the user so
select the color of the object. The range of the color selected restricting the user to have just one colored object is not a
is passed, the live video is captured and then the captured good idea. So, the model allows user to select four different
frame is flipped since it is a mirror image. The captured colors (Blue, Green, Red and Yellow) for the object color
video is then converted into HSV color range. Once the live selection.
frame appears on the screen all the color buttons and clear
button is displayed on the live frame along with the keyboard Initially when the Frame is captured it is captured in Blue
shortcuts. For the user to write in the air through their (255, 0, 0), Green (0, 255, 0), Red (0, 0, 255) that is BGR
gesture, the model will detect the object using the spacebar color space to detect the colored object easily and quickly
key. If the object is not detected a display message appears into Hue saturation value color space [2]. Various function
saying “object not within the camera range" so the user will like cv2.cvtCOLOR(), cv2.inRange(), Cv2.dilate() were used
have to bring the object within the camera range and will to converts the color-space, and create binary segmentation
have to press the space bar key again. Once the object is and to increase segmentation respectively [8]. After this the
detected color masking is done followed by the background upper Hue saturation value and lower Hue saturation value
subtraction. Next step is to find the contour. Contour is an of four different colors (Blue, Green, Red and Yellow) are
object with same color intensity and then the centre is found stored in NumPy arrays (np.array).
on the object. If contour is not detected the centre is set to C. Frame Reading
zero else the centre is found on the contour. Followed by this To capture the video in real time cv2.VideoCapture is
the user can see the output on both the screen that is the live used. This is a module of the OpenCV library. It creates a
frame and the white board. If the user wants to exit Virtual video object that would help stream or display the video. The
canvas then they will need to press esc key and the system video captured is mirror video which is then flipped by the
closes or stops running. model using cv2.flip. Also, the captured frame is in BGR
IV. SYSTEM METHODOLOGY color space which is then converted into HSV color frame
[8].
This section gives details about the different parts of the
model. Starting from IV-1 which is the Graphical User
Interface needed for selecting the object color with the help
of arrays followed by Image Segmentation that includes
frame reading and background subtraction [3], Image
Processing which is detecting the contour [4] for object
detection and finally the output which is draw on canvas that
makes use of keyboard keys. Fig.2. is the architecture of the
model

Fig.3. Whiteboard and Canvas windows

Sometimes background of live frame isn’t suitable for

displaying the written text (output) so access to a whiteboard
was necessary. The whiteboard displays the written text
(output) clearly on a white background. It is in sync with the
canvas (live window) so the delay time is 0.
This is done with the use of np.zeros which creates a
white image that acts as the background to the whiteboard.
The parameters passed to this function are the dimension of
the white image (whiteboard window). Fig.3. shows the
whiteboard along with the canvas window. The option of
Fig. 2. System Architecture resizing the frame as per requirement is done using the
CV2.Window_Normal
A. Graphic User Interface D. Canvas Creation
An interface is made using tkinter library in python. This A Canvas (live frame) is created along with a white board
interface has all details regarding what virtual canvas is and which is in sync with the canvas using the function np.zeros
its manual for easy usage of the software along with option [10]. Various feature like color buttons, clear button is
to select the object color. displayed on the canvas(live frame) as seen in Fig.4. using
B. Object Color Selection the function cv2.rectangle, cv2.putTextalong with various
functionality.
Once the user selects the Color of the object of his/her
choice then that color is passed to the model for color
masking and contour detection. This ensures that the model
detects and tracks the user specified color only.
Fig.4. Display buttons on the screen
The color of the object is detected using the Hue
saturation value which is difficult and is affected by various Five different color buttons are provided at the top center
parameters one of which is sunlight. Also, it is not easy to of the Canvas Window (live frame) which helped user to

3
Authorized licensed use limited to: HKBK College of Engineering. Downloaded on December 06,2024 at 07:47:29 UTC from IEEE Xplore. Restrictions apply.
easily switch between the five colors (Blue, Green, Red, H. Keyboard Keys
Yellow, and Pink) according to the users needs. Selected Filling the canvas (live window) with least used buttons
color appears slightly bigger than the unselect color. will occupy a lot of space which leaves with less writing
To store and handle the color points of the five colors space. So, for better user interactivity shortcuts keys were
(Blue, Green, Red, Yellow, Pink) five different arrays were provided. In total there are 6 shortcut keys, they are – (z) to
created along with color index (initially set to 0) which is undo changes, (+) to increase brush thickness,(-) to decrease
used to mark points of a specific array of that particular color brush thickness, (numbers 1 to 5) to change brush color,
[10]. The Maximum length of the color point of the colors is (s/clt s) to save and (q/esc) to quit the program. These
set as 1024. features work with the help of the keyboard module in
python. Whiteboard is in sync with the canvas (live window)
At the top left-corner of the Canvas Window (live frame) so the delay time is 0.
clear button is placed for easy access. Clear button clears all
the writing in both the screen (canvas and whiteboard) before To start writing in the air and for the model to detect the
or after saving the writing (depends on the user). object and draw points at each and every position on the
canvas and the whiteboard, user needs to press the spacebar
The five different arrays that were created to store and key. The gestures will be detected and then the output would
handle the color points of the five colors (Blue, Green, Red, be displayed.
Yellow, Pink) are all dequeued and the color index of the
five colors (Blue, Green, Red, Yellow, Pink) are initialized to
0
E. Color Masking
Color masking process creates mask of the object using
the function cv2.erode which creates a boundary of the
object detected and fills it with white [3] and then Fig.6. Three saving modes
background subtraction is performed simultaneously where
the background is changed to black using the function Fig.6. shows the save shortcut key and its 3 modes –
bgModel.apply [8]. Canvas, whiteboard, blackboard. The user can save in any of
the 3 modes. The images saved will be in PNG format at the
F. Contour Detection user’s desired location.
The model then detects the contours which is the shape of
the object found with similar HSV color intensity. This is V. EXPERIMENTS
done using cv2.findContours. As there is a possibility of A. Experiment Details
finding many contours the model will find the biggest
The model was tested in different backgrounds with
contour among all the contours found [4].
different light settings. The backgrounds included indoor and
outdoor environments. In indoor environments the model
was tested with a plain white background and with cluttered
background that included multiple colors in the background.
Three types of lighting was considered that is very bright
(brightness intensity 80%), normal (brightness intensity
40%) and very dark (brightness intensity 5%). The accuracy
rate is generated by testing the model for 5 minutes by using
all the features and different color objects.
For outdoor environments the backgrounds were a train
platform where people were moving constantly and a still
Fig.5. Contour Detection background that is in a garden with less movement in the
background. The lightings considered are very bright and
This helps in reducing the errors regarding same very dark. This is because in outdoor environments lighting
background and object color. Then with the help of cannot be adjusted. Two cameras with different Megapixels
cv2.circle a circle is formed over the object and then the are used to indentify best accuracy.
center (C) of the object is calculated as seen in Fig.5. using
the formula (1) B. Results and Discussion
The movement of the object is detected and is visible on
௠ଵ଴ ௠଴ଵ the video screen with five different colors buttons, a clear all
‫ ܥ‬ൌ ǡ (1) button and display message of the selected color at the top of
௠଴଴ ௠଴଴
Here, C = mass center the screen. Also, the whiteboard is in sync with the canvas.
m10, m01, m00 = spatial moments There are a number of options to pick from which include
various system modes that serve various functionalities.
G. Draw on Canvas Basic Functionality of the model include
After completing the above steps, the user can finally draw
x Virtual Canvas -The system will trace the
and write in the air easily and the model will track the object
movement of the object and display them in real
and will form points at each position for the air canvas
time.
effect. Then the model will display the video frame by
x Whiteboard - The system will trace the movement
frame. The function used here is imshow().
of the object and display them in whiteboard

4
Authorized licensed use limited to: HKBK College of Engineering. Downloaded on December 06,2024 at 07:47:29 UTC from IEEE Xplore. Restrictions apply.
 x Color options - The user can select a different color From the experiments conducted we can conclude that
for the text from the available options. the model gives accurate results in plain background whether
x Clear all – Clear all clears the characters on the that is in indoor or outdoor environments. The quality of the
screen and provides a clean space to write on. camera plays a very important role for accuracy when
background is cluttered. For outdoor environments with
The model gives different accuracy rates when placed in crowded background the accuracy can be increased if the
different backgrounds. Tables 1 and 2 give a view of the camera quality is increased.
accuracy rates in different environments. VI. CONCLUSION
TABLE I. ACCURACY RATES FOR INDOOR ENVIRONMENTS The user is able to draw and write on the screen with ease
on a virtual canvas, allowing them to effectively interact with
Test Camera Back- Lighting Accuracy others. This technology has the ability to put standard writing
No. Quality ground (%) methods to the test. We covered various types of
1 1.3 MP Plain Very 94%
technologies that could be employed to construct this system.
White Bright
Virtual canvas works well with a good quality camera as
2 1.3 MP Plain Normal 98%
well as a clear background. Few challenges were faced if the
White
room is exposed to too much sunlight or in cluttered
3 1.3 MP Plain Very 91%
White Dark background which results in weak object color detection .
4 1.3 MP Cluttered Very 90% Excluding these challenges Virtual canvas strikes to be an
(many Bright excellent software for interacting with the digital
objects) environment. This model can now produce the output that
5 1.3 MP Cluttered Normal 98% our group had envisioned, and it will be enhanced in the
(many future to include new features and improve efficiency.
objects)
6 1.3 MP Cluttered Very 86% REFERENCES
(many Dark [1] Rajesh D, Yamini C “A Review on Object Detection using OpenCV”
objects) in International Journal of Innovative Technology and Exploring
Engineering (IJITEE)
7 5 MP Cluttered Very 94%
(many Bright [2] Suraj S, Shende, Vikas S, Bharti P “Real Time Object Detection
using Deep-Learning and OpenCV” in International Research Journal
objects) of Engineering and Technology (IRJET)
8 5 MP Cluttered Very 92%
[3] Mohamed, Shahrizat Shaik, Nooritawati Md Tahir, and Ramli Adnan.
(many Dark “Background modelling and background subtraction performance for
objects) object detection.” 2010 6th International Colloquium on Signal
Processing & its Applications. IEEE, 2010.
TABLE II. ACCURACY RATES FOR OUTDOOR ENVIRONMENTS [4] Raghav P, ArchitG , Manas S, in “Contour, Shape & Color Detection
using OpenCV-Python” in International Journal of Advances in
Test Camera Back- Lighting Accuracy Electronics and Computer Science
No. Quality ground (%) [5] Mitra, Sushmita. Senior Member, IEEE, and Tinku Acharya, Senior
1 1.3 MP Crowded Very 81% Member, IEEE,“Gesture Recognition: A Survey”. IEEE Transactions
(People Bright On Systems, Man, And Cybernetics—part C: Applications And
walking) Reviews 37.3 (2007): 311.
2 1.3 MP Crowded Very 84% [6] Chen, Mingyu, Ghassan AlRegib, and Biing-Hwang Juang. “Air-
(People dark writing recognition—Part I: Modeling and recognition of characters,
walking) words, and connecting motions.” IEEE Transactions on Human-
Machine Systems 46.3 (2015): 403-413.
3 5 MP Crowded Very 89%
(People Bright [7] Chen, Mingyu, Ghassan AlRegib, and Biing-Hwang Juang. “Air-
writing recognition—Part II: Detection and recognition of writing
walking)
activity in continuous stream of motion data.” IEEE Transactions on
4 5 MP Crowded Very 91% Human-Machine Systems 46.3 (2015): 436-444.
(People dark [8] Vandit G “Real Time Color Segmentations and Color Filtration” in
walking) IJSRD National Conference on Advances in Computer Science
5 1.3 MP Plain Very 94% Engineering & Technology May 2017
Bright [9] D. U. Lakshmi and B. Harish, "A novel air writing recognition system
6 1.3 MP Plain Very 91% using Raspberry Pi for the control and interaction of digital systems,"
Dark 2017 International Conference on Energy, Communication, Data
7 5 MP Plain Very 94% Analytics and Soft Computing (ICECDS), 2017
Bright [10] S.U. Saoji, Nishtha D, Akash C, B Phogat “Air canvas application
8 5 MP Plain Very 92% using Open CV and numpy in python” in (IRJET) Volume: 08 Issue:
08 | Aug 2021
dark

5
Authorized licensed use limited to: HKBK College of Engineering. Downloaded on December 06,2024 at 07:47:29 UTC from IEEE Xplore. Restrictions apply.