An Effective Card Scanning Framework for User

Authentication System
Hania Arif Ali Javed
Department of Software Engineering Department of Software Engineering
University of Engineering and Technology University of Engineering and Technology
Taxila, Pakistan Taxila, Pakistan
[email protected] [email protected]

Abstract—Exponential growth of fake ID cards generation identification in real-time using these biometric solutions is a
leads to increased tendency of forgery with severe security and challenging task due to variation in human pose [6], facial
privacy threats. University ID cards are used to authenticate expressions [7], hair styles [8], illumination conditions [9],
actual employees and students of the university. Manual user distance from the camera [10], etc. These conditions
examination of ID cards is a laborious activity, therefore, in this
paper, we propose an effective automated method for
affect the identification process and leads to the generation of
employee/student authentication based on analyzing the cards. false alarms.
Additionally, our method also identifies the department of There is a common practice of identifying users through
concerned employee/student. For this purpose, we employ
employee card verification. Employee card verification is
different image enhancement and morphological operators to
improve the appearance of input image better suitable for largely used in various organizations in Pakistan for user
recognition. More specifically, we employ median filtering to identity verification [11]. Currently, organizations either used
remove noise from the given input image. Next, we apply the RFID card scanners [13] or manual card verification for user
histogram equalization to enhance the contrast of the image. We authentication. Manual card verification is a taxing and time-
employ Canny edge detector to detect the edges from this consuming activity and become less effective in case of large
equalized image. The resultant edge image contains the broken number of users entering in a building in short time span.
characters. To fill these gaps, we apply the dilation operator that Many educational institutions prefer the use of simple
increases the thickness of the characters. Dilation fills the identification cards without RFID that the employee or student
broken characters, however, also add extra thickness that is
has to wear as a part of their dress code in order to reduce the
then removed through applying the morphological thinning.
Finally, dilation and thinning are applied in combination to cost of user identification. However, this manual verification
Optical character recognition (OCR) to segment and recognize process is unreliable and leads to security loop holes due to
the characters including the name, ID, and department of the reliance on guards for manual verification where image
employee/student. Finally, after the OCR application on the tampering can be used to generate false cards that are hard to
morphed image, we obtain the name, ID, and department of the detect from naked eye.
employee/student. If the concerned credentials of the
employee/student are matched with his/her department, then To address these challenges, we propose an automated
access of the door is granted to that employee/student. low-cost solution to authenticate the identity of the
Experimental results illustrate the effectiveness of the proposed employees/ students. More specifically, we analyze the
method. captured frames of the card via cameras and employ OCR to
detect the department of the concerned person. We propose a
Keywords—Dilation, Image Enhancement, Morphological non-learning-based technique to reduce the computational
Thinning, Optical Character Recognition, User Authentication. complexity of our solution. We employ different enhancement
I. INTRODUCTION and morphological operators to obtain the image better suited
for OCR processing. This image is finally fed to the OCR
The advancement in modern day sophisticated image algorithm that recognize the employee’s identity and
forgery algorithms led to increased tendency of image department for authentication purposes. The main
manipulation that poses several security and privacy issues. Contributions of the proposed work are as follows:
Organizations security concerns regarding confidentiality of
their employee’s working environment led to the usage of • We propose an effective user identification system
various automated solutions to testify the entrance of through applying different morphological and image
authorized employees in relevant departments/buildings. enhancement techniques.
Organizations frequently use various devices such as
• We present an economical and efficient user
surveillance cameras [1], lock systems [1], RFIDs [2], [3], iris
authentication system where a low-cost camera is
recognition systems [3], [4], fingerprint recognition systems
required to capture the video.
[5], etc. for employees’ authentication. However, the current
state-of-the-art image processing and artificial intelligence • Our system is robust to variations in illumination
algorithms can be used to exploit the authentication systems. conditions, glare, noise, contrast, and shadows.
To address this issue, there exists a dire need to propose more
effective and secure authentication systems that are robust to II. LITERATURE REVIEW
these forged/manipulated input data (i.e. images, videos, etc.). This section provides a critical analysis of existing state-
There is a common practice of using different biometric of-the-art user authentication systems. Existing techniques
solutions [8] – [12] i.e. thumb print identification, facial have proposed either biometric systems [14] – [20] or RFID
recognition, iris recognition, etc. for user authentication. based systems [22] – [25] for user authentication.
These solutions are effectively used by organizations to Multiple biometric identification techniques make use of
authenticate the identity of their employees. However, user the waves emerging out of the human body for human

978-1-7281-4235-7/20/$31.00 ©2020 IEEE

biometric identification and authorization. For example, [14]
– [16] makes use of EEG signals coming out of brain for this
purpose which are then classified using neural network
algorithm for user classification. I. Assadi, A. Charef, N.
Belgacem, A. Nait-Ali, and T. Bensouici [14] used the ECG
signals which are classified using K-Nearest Neighbors
Classifier. Behavioral biometrics-based methods have also
been proposed by researchers [18] – [20] in order to identify
and authenticate the users. Y. Chemla and C. Richard [22]
used smart card along with biometric profile recognition
method for personal identification. The method in [21]
includes the use of client-server-based architecture that helps
in identifying and matching the smart card holder with his
biometric data, making the system fool proof.
Existing systems have used both deep learning and
conventional machine learning algorithms [26] – [29] for user
identification and classification. Deep learning approaches are
employed due to potential benefits of achieving better
accuracy [30], [31]. Additionally, existing literature also
achieved better accuracies using non-learning-based
techniques. For example, A. Nosseir and O. Adel [32] used
SUFE extraction algorithm along with template matching
technique to identify a person from his ID card in multiple
environmental conditions. Fig. 1. Block Diagram of Proposed System
OCR based algorithms are also now-a-days commonly
used by many firms and researchers for user identification. For (i) (i)
IM (x,y) = median{IGS (x,y),(x,y) ∈ w} ()
example, C. Wick, C. Reul, and F. Puppe [33] proposed an
OCR based algorithm using LSTM based network. On the (i)
other hand, R. Baran, P. Partila, and R. Wilk [34] proposed a where IM (x,y) is the resultant median image obtained after
non-learning-based OCR method. This method used applying median filtering and w is the size of window that is
connected component labelling algorithm to reduce the set to 3x3 in our case.
distortions in the input image and later fed to OCR for
authentication. C. Histogram Equalization
We observed that the images captured at low illumination
III. PROPOSED METHOD conditions also experience poor contrast. Therefore, after
This section provides a comprehensive discussion on the noise reduction, we employed the histogram equalization
proposed method. Shown in Fig. 1 is the process flow of the method to improve the contrast of the image. The histogram
proposed method. equalization process enhances the image contrast by adjusting
the image intensity distribution.
A. Image Pre-processing
More specifically, we applied the global histogram
We used the 16MP resolution camera to capture the input equalization method for contrast enhancement as follows:
color frame. The pre-processing stage transforms the acquired
color image into grayscale image as follows: (i)
Ik = C × ∑kj=0 (nj / N), for 0 ≤ k ≤ L-1 ()
(i) (i) (i)
IGS (x,y) = IR (x,y) × 0.298 + IG (x,y) × 0.587 (i)
(i)
+ IB (x,y) × 0.114 () where Ik ,nj , N, C and L represent the new intensity value,
frequency of intensity j, sum of all frequencies up to intensity
(i) (i) k, constant value, and highest frequency respectively.
where IGS (x,y) represents the grayscale image, IR (x,y),
(i) (i)
IG (x,y) and IB (x,y) represent the red, green, and blue D. Edge Detection
components of the input color image respectively. After contrast enhancement, we transform this image into
edge image to extract the text from the image. For this
B. Noise Removal
purpose, we employ the canny edge detector as canny operator
We observed after watching massive number of images in extracts maximum edges as compared to other edge detection
our dataset that card images contain spiky black dots/patches techniques. Additionally, canny edge detector retains
due to placing cards in transparent covers. These patches important information by keeping all important major and
generate impulse noise in the captured images. Median minor edges while discarding rest of the unnecessary details
filtering is most effective method to reduce the density of in the image. We obtain the edge image as follows:
impulse noise in the images as it also preserves the edges of
the image. Therefore, in this paper, we employed the median (i) (i)
filtering to reduce the impulse noise from the image as Iedge (x,y) = canny{IHE (x,y),(x,y) ∈ w} ()
follows:
E. Dilation entire frame to the OCR method without manual image
The edge image obtained in the last step contains broken cropping to specify any region of interest (ROI). The results
characters or gaps that must be filled before feeding image to obtained from OCR are then compared with the departmental
OCR. Therefore, we applied the dilation operator on the edge data. If the department of the employee matches with the
image as follows: record in the database, then that employee is authorized to
enter from the door.
(i) (i)
Idil (x,y) = Iedge (x,y) ⊕ SE () IV. PERFORMANCE EVALUATION

(i) A. Dataset
where Idil (x,y) represents the dilated image and SE represents
For the performance evaluation of the proposed system,
the structuring element which is set to a window of size 3 for
we created a dataset of 1000 university card images that
both horizontal and vertical dilation.
belong to different departments. We ensured to create a
Dilation operation successfully fills the broken characters; diverse dataset. Our dataset is captured in different
however, thickness of the characters is increased significantly. environmental and lighting conditions, containing glare,
Therefore, we need to reduce the extra thickness of characters shadow, and low contrast images. The resolution of each
added after dilation. image in the dataset is 3120 x 4160 pixels.
F. Density Reduction B. Experimental Results
The density reduction stage removes the extra thickness of Performance of the proposed system is evaluated in images
the edges by applying the morphological thinning operation. taken at a real-time. The effectiveness of the proposed system
Thinning operator is used to truncate the outliers. After is evaluated by the detection of correct credentials of the
applying the thinning operator on the dilated image, we obtain employee/student. Shown in Table I are the results of the
the characters with actual thickness. Morphological thinning proposed system for user identification. From the results
is applied as follows: (Table I), we can observe that our system achieves an average
accuracy of 96.4%. We captured the frames for testing in real-
(i) (i)
Ithin (x,y) = Idil (x,y) ⊗ SE () time environmental conditions with poor contrast, low
illumination, shadow over image, background distortion and
(i) glare. Despite the presence of multiple distortions in the
where Ithin (x,y) represents the thinned image. We used SE of captured images, our system provides remarkable
square shape which had the size equal to a 3x3 window. performance. It is worth mentioning that the system gives
G. Hole Filling greater than 95% accurate results for six environmental
conditions out of total 7 conditions under consideration. Tilt
After density reduction, we observed that the body of in the scanned image affects the system accuracy due to the
characters contains small holes that need to be filled. We fact that the proposed system is designed to scan images from
achieved hole filling step as follows: an anterior view with the angle of inclination of camera
(i) (i-1) (i) c
parallel to the plane of identification card.
Ifilled = (Ifilled ⊕ SE) ∩ Ithin ()
In our second experiment, we perform a confusion matrix
(i) (i) c
analysis for user authentication based on different
where Ifilled and Ithin are
the filled image and complement of departments. The results of confusion matrix analysis are
thinned image respectively. provided in Table II. From the results presented in Table II,
we can observe that the proposed system achieves 100% true
H. Optical Character Recognition
positives for seven departments out of total 11 classes. In the
To recognize the contents of the card, we apply the remaining four categories, the highest error is just 0.2 that
tesseract OCR method [35] on this filled image obtained after signify the effectiveness of our system in terms of user
applying the morphological operators. We obtain the authentication based on automated card verification. Hence,
recognized characters and use spacing detection mechanism we can argue that the proposed system is effective in terms of
to extract different words. Later, we convert the recognized classifying the employee/student as an authorized or
characters/words into computerized characters. We fed the unauthorized personnel. The results also show that the system
TABLE I. EMPLOYEE/ STUDENT UNIVERSITY CARD IDENTIFICATION RESULTS

True True False False Precision Recall Accuracy Error F1

Image Type
Positive Negative Positive Negative Rate Rate Rate Rate Score

Poor Contrast 49 5 0 1 100% 98% 98.18% 1.82% 0.9899

Low Illumination 30 3 0 0 100% 100% 100% 0% 1
Shadow 33 5 0 2 100% 94.29% 95% 5% 0.9706
Background Distortion 50 5 0 0 100% 100% 100% 0% 1

Glare 49 3 0 1 100% 98% 98.11% 1.89% 0.9899

Slight Tilt 8 2 1 1 88.89% 88.89% 83.33% 16.67% 0.8889
Normal Image 45 10 0 0 100% 100% 100% 0% 1
Average 98.41% 97.02% 96.37% 3.63% 0.9770
 TABLE II. CONFUSION MATRIX ANALYSIS FOR DEPARTMENT BASED IDENTIFICATION

Departments / Category SE IE CE EE ME ENC ENV CS CP TE X

Software Engineering (SE) 1 0 0 0 0 0 0 0 0 0 0

Industrial Engineering (IE) 0 0.9 0 0 0 0 0 0 0 0 0.1

Civil Engineering (CE) 0 0 1 0 0 0 0 0 0 0 0
Electrical Engineering (EE) 0 0 0 1 0 0 0 0 0 0 0

Mechanical Engineering (ME) 0 0 0 0 1 0 0 0 0 0 0

Electronics Engineering (ENC) 0 0 0 0 0 0.8 0 0 0 0 0.2
Environmental Engineering (ENV) 0 0 0 0 0 0 0.9 0 0 0 1

Computer Science (CS) 0 0 0 0 0 0 0 1 0 0 0

Computer Engineering (CP) 0 0 0 0 0 0 0 0 0.9 0 1
Telecom Engineering (TE) 0 0 0 0 0 0 0 0 0 1 0

Unauthorized (X) 0 0 0 0 0 0 0 0 0 0 1

REFERENCES
Normal Image Poor Contrast [1] M. Schiefer, “Smart home definition and security threats,” in 2015
Slight Tilt Low Illumination ninth international conference on IT security incident management &
Glare Shadow IT forensics, 2015, pp. 114–118.
Background Distortion
1 [2] S. Bauk and A. Schmeink, “RFID and PPE: Concerning workers’
safety solutions and cloud perspectives a reference to the Port of Bar
0.8 (Montenegro),” in 2016 5th Mediterranean Conference on Embedded
True Positive Rate

Computing (MECO), 2016, pp. 35–40.

0.6 [3] J. Xu, H. Gao, J. Wu, and Y. Zhang, “Improved safety management
system of coal mine based on iris identification and RFID technique,”
0.4 in 2015 IEEE International Conference on Computer and
Communications (ICCC), 2015, pp. 260–264.
0.2 [4] S. Barra, A. Casanova, F. Narducci, and S. Ricciardi, “Ubiquitous iris
recognition by means of mobile devices,” Pattern Recognit. Lett., vol.
0 57, pp. 66–73, 2015.
0 0.2 0.4 0.6 0.8 1 [5] C. Yuan, X. Sun, and R. Lv, “Fingerprint liveness detection based on
False Positive Rate multi-scale LPQ and PCA,” China Commun., vol. 13, no. 7, pp. 60–65,
2016.
Fig. 2. ROC Curve for University Card Identification [6] X. Sun, J. Shang, S. Liang, and Y. Wei, “Compositional human pose
regression,” in Proceedings of the IEEE International Conference on
is independent of the changes in major environmental Computer Vision, 2017, pp. 2602–2611.
condition. [7] C. L. Witham, “Automated face recognition of rhesus macaques,” J.
Neurosci. Methods, vol. 300, pp. 157–165, 2018.
In the last experiment, performance of the proposed [8] M. Shirodkar, V. Sinha, U. Jain, and B. Nemade, “Automated
system is measured using the receiver operating curve (ROC) attendance management system using face recognition,” Int. J.
analysis. We created ROC curves for images acquired in Comput. Appl., vol. 975, p. 8887, 2015.
multiple conditions and results are plotted in Fig. 2. From the [9] W. Zhao and R. Chellappa, “Image-based face recognition: Issues and
methods,” Opt. Eng. York-marcel dekker Inc., vol. 78, pp. 375–402,
Fig. 2, it is evident that the proposed system attains 2002.
remarkable classification accuracy for all conditions except [10] M. Ao, D. Yi, Z. Lei, and S. Z. Li, “Face recognition at a distance:
the tilted acquired images. system issues,” in Handbook of Remote Biometrics, Springer, 2009,
pp. 155–167.
V. CONCLUSION [11] C. J. Bennett and D. Lyon, Playing the identity card: surveillance,
In this paper, we propose an effective and efficient security and identification in global perspective. Routledge, 2013.
employee authentication method based on card verification. [12] M. A. Sarrayrih and M. Ilyas, “Challenges of online exam,
The proposed method is robust to variations in illumination performances and problems for online university exam,” Int. J.
Comput. Sci. Issues, vol. 10, no. 1, p. 439, 2013.
conditions, glare, noise, contrast, and shadows. Additionally,
[13] S. Ravichandran, “Smart Identity Card.”
our method is economical and require any low-cost camera for
[14] I. Assadi, A. Charef, N. Belgacem, A. Nait-Ali, and T. Bensouici,
video/image capturing. Performance of the proposed method “QRS complex based human identification,” in 2015 IEEE
is evaluated on a diverse dataset of real time scanned images. International Conference on Signal and Image Processing Applications
Our method achieves an average accuracy of 96% that (ICSIPA), 2015, pp. 248–252.
demonstrates the effectiveness of the proposed method for [15] Page, Adam, Amey Kulkarni, and TinooshMohsenin. "Utilizing deep
employee authentication. Under the condition of slight tilt in neural nets for an embedded ECG-based biometric authentication
image acquisition process, the accuracy drops to some extent. system." 2015 IEEE Biomedical Circuits and Systems Conference
(BioCAS). IEEE, 2015.
Currently, we are examining this problem and planning to
extend our method that can also achieve remarkable [16] Kaur, Barjinder, Dinesh Singh, and ParthaPratim Roy. "A novel
framework of EEG-based user identification by analyzing music-
performance under tilt image acquisition condition. listening behavior." Multimedia tools and applications 76.24 (2017):
 25581-25602. arXiv1907.12145, 2019.
[17] Gui, Qiong, ZhanpengJin, and Wenyao Xu. "Exploring EEG-based [27] D. Cheng, Y. Gong, S. Zhou, J. Wang, and N. Zheng, “Person re-
biometrics for user identification and authentication." 2014 IEEE identification by multi-channel parts-based cnn with improved triplet
Signal Processing in Medicine and Biology Symposium (SPMB). loss function,” in Proceedings of the iEEE conference on computer
IEEE, 2014. vision and pattern recognition, 2016, pp. 1335–1344.
[18] Bailey, Kyle O., James S. Okolica, and Gilbert L. Peterson. "User [28] Z. Wu, Y. Huang, L. Wang, X. Wang, and T. Tan, “A comprehensive
identification and authentication using multi-modal behavioral study on cross-view gait based human identification with deep cnns,”
biometrics." Computers & Security 43 (2014): 77-89. IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 2, pp. 209–226,
[19] Bo, Cheng, et al. "Silentsense: silent user identification via touch and 2016.
movement behavioral biometrics." Proceedings of the 19th annual [29] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature
international conference on Mobile computing & networking. ACM, learning approach for deep face recognition,” in European conference
2013. on computer vision, 2016, pp. 499–515.
[20] Alzubaidi, Abdulaziz, and Jugal Kalita. "Authentication of smartphone [30] J. Zhu, H. Ma, J. Feng, and L. Dai, “ID card number detection
users using behavioral biometrics." IEEE Communications Surveys & algorithm based on convolutional neural network,” in AIP Conference
Tutorials 18.3 (2016): 1998-2026. Proceedings, 2018, vol. 1955, no. 1, p. 40124.
[21] W. B. Lund, D. J. Kennard, and E. K. Ringger, “Combining multiple [31] N. Wang, X. Zhu, and J. Zhang, “Research of ID card recognition
thresholding binarization values to improve OCR output,” in Document algorithm based on neural network pattern recognition,” in 2015
Recognition and Retrieval XX, 2013, vol. 8658, p. 86580R. International Conference on Mechatronics, Electronic, Industrial and
[22] Y. Chemla and C. Richard, “Security device, method and system for Control Engineering (MEIC-15), 2015.
financial transactionas, based on the identification of an individual [32] A. Nosseir and O. Adel, “Automatic Extraction of Arabic Number from
using a biometric profile and a smart card.” Google Patents, 08-Aug- Egyptian ID Cards,” in Proceedings of the 7th International Conference
2017. on Software and Information Engineering, 2018, pp. 56–61.
[23] Hameed, Sarmad, et al. "Radio frequency identification (RFID) based [33] C. Wick, C. Reul, and F. Puppe, “Improving OCR Accuracy on Early
attendance & assessment system with wireless database records." Printed Books using Deep Convolutional Networks,” arXivPrepr.
Procedia-Social and Behavioral Sciences 195 (2015): 2889-2895. arXiv1802.10033, 2018.
[24] Zaman, Hasan U., et al. "RFID based attendance system." 2017 8th [34] R. Baran, P. Partila, and R. Wilk, “Automated text detection and
International Conference on Computing, Communication and character recognition in natural scenes based on local image features
Networking Technologies (ICCCNT). IEEE, 2017. and contour processing techniques,” in International Conference on
[25] Jackson, Daniel, Fred Bargetzi, and Brian Donlan. "User identification Intelligent Human Systems Integration, 2018, pp. 42–48.
and location determination in control applications." U.S. Patent No. [35] Smith, Ray. "An overview of the Tesseract OCR engine." In Ninth
9,602,172. 21 Mar. 2017. International Conference on Document Analysis and Recognition
[26] S. Homayon and M. Salarian, “Iris recognition for personal (ICDAR 2007), vol. 2, pp. 629-633. IEEE, 2007.
identification using LAMSTAR neural network,” arXivPrepr.