Received November 26, 2019, accepted December 19, 2019, date of publication December 25, 2019,

date of current version January 7, 2020.

Digital Object Identifier 10.1109/ACCESS.2019.2962315

Image-Based Service Recommendation System:

A JPEG-Coefficient RFs Approach
FARHAN ULLAH 1, BOFENG ZHANG 1, AND REHAN ULLAH KHAN 2,3
1 School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China
2 Department of Information Technology, College of Computer, Qassim University, Buraydah 51452, Saudi Arabia
3 Intelligent Analytics Group (IAG), College of Computer, Qassim University, Buraydah 51452, Saudi Arabia

Corresponding author: Bofeng Zhang ([email protected])

This work was supported by the National Key R&D Program of China under Grant 2017YFC0907505.

ABSTRACT Online shopping platforms are growing at an unprecedented rate all over the world. These
platforms mostly rely on search engines, which are still primarily based on the knowledge-base and use
keywords matching for finding similar products. However, customers want an interactive setup that is
convenient and reliable for querying related products. In this paper, we propose a novel idea of searching for
products in an online shopping system using an image-based approach. A user can provide, select, or click
an image, and similar image-based products will be presented to the user. The proposed recommendation
system is based on content-based image retrieval and is composed of two major phases; Phase 1 and
Phase 2. In Phase 1, the proposed approach learns the class/type of the product. In Phase 2, the proposed
recommendation system retrieves closely matched similar products. For Phase 1, the proposed approach
creates a model of products using Machine Learning (ML). The model is then used to find the category of
the test products. From the ML perspectives, we employ the Random Forests (RF) classifier, and for feature
extraction, we use the JPEG coefficients. The dataset used for proof of concepts includes 20 categories of
products. For image-based recommendation, the proposed RF model is evaluated for Phase 1 and Phase 2.
In Phase 1, the evaluation of the proposed model generates a 75% accurate model. For performance
enhancements, the RF model has been integrated into the Deep Learning (DL) setup achieving 84% accurate
predictions. Based on the custom evaluation approach for Phase 2, the proposed recommendation approach
achieves 98% correct recommendations, thus demonstrating its efficacy for the product recommendation in
practical applications.

INDEX TERMS Recommendation system, services, machine learning, random forests, deep learning, SVM.

I. INTRODUCTION with a convenient and comfortable approach. Almost every

The online retail websites are rapidly evolving and their E-commerce enterprise nowadays optimize its recommenda-
popularity is exponentially growing. As in the Nielsen Global tion system. The optimal recommendation system will not
Connected Commerce Survey (2015) [22], 63% of respon- only benefit the customers but will also help the retailer by
dents who bought the products during the past six months increasing sales and customer satisfaction.
stated that they used online services. Online shoppers face Traditional E-commerce based search engines are still
the problem of selecting products from a large number of struggling for an optimal recommendation system because
available online products. The number of these products is most of these services employ text-based searching. Most
exponentially increasing due to a large number of companies of these systems use textual meta-data of products such as
moving to online businesses. This business growth is useful attributes, descriptions, and purchase histories of the different
on the one hand, but on the other hand, it creates a problem users. The textual and key-words based systems can mis-
for the user to accurately and optimally buy the desired lead the recommendation system for the inferior quality and
item. Thus these challenges and concerns have raised the non-related products needed by the users. Recently, machine
demands for recommendation systems to facilitate the users learning algorithms and models have shown its efficacy in
detection and recognition scenarios, for example, Neural
The associate editor coordinating the review of this manuscript and Networks (NN), Decision Trees (DT), and Deep Learning
approving it for publication was Joey Tianyi Zhou. (DL). Therefore, the traditional search paradigms of the text

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/
3308 VOLUME 8, 2020
F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

description can be replaced or augmented by the visual search The rest of the paper is organized as follows: Section 2 dis-
for the product recommendation system. A picture of any cusses previous related work. Section 3 discusses the ML
product should clarify the user’s demands for the appearance, models and JPEG features. Section 4 explains the proposed
usage, and brand of the desired products. While the computer approach in the form of Phase 1 and Phase 2. Section 5 eval-
vision and image processing fields have matured, the appli- uates the Phase 1 and Phase 2 of the image-based recom-
cations of product retrieval based on the image features using mendation approach and presents the comparative analysis.
artificial intelligence in the online shopping domain remains Section 6 concludes the paper.
mostly unexplored and is open to new findings.
In this paper, based on user interaction and interests, II. ILITERATURE REVIEW
we propose a novel idea to search the products efficiently The authors in [1] present a novel approach for One-Class
in an online shopping system using image-based search- collaborative filtering that is based on estimating the users’
ing techniques. As such, a user provides or selects an fashion-aware personalized ranking. The recommendation
image, and similar products (images) are presented to the system combines high-level visual features extracted by the
user. The proposed recommendation system consists of two Convolutional Neural Network (CNN) from the past feedback
major phases: In Phase 1, the proposed approach learns the and evolving trends. In [2], the proposed approach models the
class/type of the product based on the image characteristics. human sense of the relationships between objects based on
In Phase 2, the proposed recommendation system retrieves their appearances. The approach is based on the human per-
closely matched similar products. ception of visual connections between products. This human
For Phase 1, the proposed approach takes advantage of ML perception is modeled as a network inference graph problem.
for learning the features of the image/product and generates It is thus capable of recommending clothes and accessories
a learned model. The model is then used to find the cate- together with excellent subjective performance. For image-
gory/class of query products. Once the category/type of the based recommendation, the authors in [4] aim to recommend
product is identified, in Phase 2, the JPEG feature vectors- images using Tuned Perceptual Retrieval (TPR), Comple-
based Euclidian distance is used to retrieve the top 20 match- mentary Nearest Neighbor Consensus (CNNC), Gaussian
ing products from the available items in the particular class Mixture Models (GMM), Markov Chain (MCL), and Tex-
of products. These 20 items are then further processed by ture Agnostic Retrieval (TAR). The authors report that the
the proposed ‘‘Struct-Hist’’ approach for retrieving the top CNNC, GMM, TAR, and TPR easy to train; however, CNNC
10 most relevant products. The Struct-Hist uses the image and GMM are complex, while the TPR, GMM, and TAR
feature and is explained in the corresponding section. do not generalize well. In [5], the authors apply the Alex-
From the ML phase in Phase 1 of a product’s class learning, Net [32] based DL that is used to model one thousand dif-
we propose the Random Forests (RF) meta-classifier due ferent categories of images. The authors in [6] use the CNN
to its generalization capabilities and excellent performance model which classifies images into their relevant classes
in state of the art. For learning the class of the products in the ImageNet Challenge. Image similarity has also been
and the feature extraction from images, we use the JPEG investigated in state of the art as in [7] and [8]. However,
coefficients as image features. The dataset used for proof of most of the approaches are based on category similarity,
concept contains product images and labels from the Amazon i.e., the products are similar if they are in a similar category.
website. This dataset includes images having 20 categories However, there is a possibility that the images might belong
of products. For image-based recommendation, the proposed to a different category. In such cases, machine learning helps
RF model is evaluated for Phase 1 and Phase 2. In Phase 1, in finding the classes of the images. Therefore, for category
the evaluation of the proposed model generates a 75% accu- retrievals, one approach is to first classify images into their
rate model. For performance enhancements, the RF model is respective categories. Once the category of a particular image
further integrated into the DL setup and achieves 84% accu- is identified, the next step recommends the images from
rate predictions. Based on the custom evaluation approach that category. The article [9], uses the NN to calculate the
for phase 2, the proposed recommendation approach delivers similarities within the category. In [12], the authors focus
98% correct recommendations and demonstrates its efficacy on learning similarity by using CNN for multi-products in
for the recommendation of image-based systems. a single image. In [13], the authors concentrate on image
From the implementation point of view of the recom- similarity and image semantics. The authors in [14] adopt the
mendation system, a user submits or selects an image of image descriptions of visual denotations. They propose a new
the product, and similar products (images) are presented to similarity metrics of semantic inference for event descrip-
the user. The proposed approach is based on this assump- tions. Similarly, the article [15] proposes the semantic image
tion. However, this assumption also covers the use-cases browser for representing the information visualization with
in which the query image is obtained from the user selec- the automated intelligent image analysis. The authors in [23]
tion, clicking, and history of user purchases. Thus, our pro- address the issue of long queries and propose the contextual-
posed approach is based on learning features from user-based based image recommendation system. In [24], the authors
images and recommending similar products based on these use Geo-tagged images from social media for the travel rec-
images. ommendation system. They integrate the time, location, and

VOLUME 8, 2020 3309

 F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

weather information with the recorded images. The authors

in [25] take advantage of the Geo-tagged images for road-
based travel recommendation system. The proposed approach
recommends the most famous landmarks and travel routings.
The authors use the Spatial Clustering (SC) approach for
ranking and identifying the landmarks. In [26], the authors FIGURE 2. The example of 8 × 8 pixel JPEG block matrix.
discuss the content-based based approach for image-based
recommendation retrieval. The article [27] recommends a
personality-based recommendation based on the personality C. JPEG-COEFFICIENT FEATURES
traits in images. JPEG Coefficient algorithm is developed by the Joint Pho-
tographic Expert Group (JPEG). The JPEG algorithm con-
III. CLASSIFICATION AND FEATURES verts the original input image into the YCbCr color space
In this section, we discuss the classification paradigm and the for compression purposes. The objective is reducing the size
features set used for the recommended approach. of the image while maintaining optimal quality. The image
is divided into 8 × 8 or 16 × 16 pixel blocks, as shown
A. RANDOM FOREST (RF) in Figure 2. Discrete Cosine Transform (DCT) is applied to
An RF algorithm by Leo Breiman [17] is a supervised algo- the 8 × 8 pixel window which generates 64 values. Then by
rithm that generates forests having many trees. In the RF the JPEG quantization process, the high-frequency values of
classifier approach, generally, if we have more trees in the for- an image are discarded, and the low-frequency details are
est, the higher is the performance of the classification model. preserved. Before the recommendation task, we extract the
RF model belongs to the ensemble classifier models, which JPEG image features of each image in our data set. For feature
are primarily adapted for problems including classification extraction and feature vector construction, we apply the JPEG
and regression. The RF algorithm operates by producing Coefficient algorithm, which is represented as follows.
decision trees in the training phase. These trees output the M −1 N −1
class labels, and the final decision of the class depends on the X X π (2m + 1) p π (2n + 1) q
Bpq = αp αq Amn cos cos ,
majority voting of the trees. Figure 1 shows the flow of the RF 2M 2N
m=0 n=0
classifier. A sample ‘‘n’’ from ‘‘N’’ samples is input to the RF 0≤p≤M −1
classifier. The RF model first generates several trees by using
feature subsets. The individual tree presents a classification 0≤q≤N −1 (1)
result. The result of the classification depends on the majority where, the parameters αp and αq are as follows:
voting system. The test sample is assigned to the class that 
1

1
gets the highest voting scores from the trees. √ , p=0

 √ , q=0


αp = r M αq = r N
2 2
, 1 ≤ p ≤ M −1 , 1 ≤ q ≤ N −1

 

 
M N
(2)
Thus the αp and αq are normalized scaling factors for
orthonormal transformation, P is the horizontal spatial fre-
quency, Q is the vertical spatial frequency. Amn is the pixel
value at coordinates (m, n). Bpq is the DCT coefficient at
coordinates (p, q).

IV. PROPOSED MODEL

FIGURE 1. A generic representation of RF classifier. From the implementation point of view of the recommen-
dation system, a user submits or selects an image of the
B. CONVOLUTION NEURAL NETWORKS (CNN) product, and similar products/images are presented to the
CNN has proved to be a handy and innovative tool of DL user. This is an assumption of the proposed approach. How-
to learn image feature sets and inherent relationships in low- ever, this assumption also covers the use-cases in which the
level features to higher-level objects in images. The generic query image is obtained from the user selection, clicking,
architecture of CNN contains interconnected layers. The and history of user purchases. Thus, the proposed approach
CNN has repeated convolutional blocks, Rectified Linear is based on learning features from user-based images and
Units (ReLU), and pooling layers [32]. Convolutional layers recommending similar products based on these images. This
perform the convolution of the input with a set of filters. The assumption synchronizes us with the image-based content
filters learned during the training phase. The nonlinearity in recognition paradigm of Computer Vision and ML. The pro-
the data is modeled by the ReLU layer [32]. The pooling layer posed recommendation system based on the assumption is
samples the input and consolidates the image class. shown in Figure 3 and is composed of two major phases:

3310 VOLUME 8, 2020

F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

FIGURE 4. The first phase of the recommendation process. The first phase
learns the features using the ML classifier to determine the
category/class of the products based on the image features.

FIGURE 3. The generic flow of the proposed recommendation

approach. Green represents ‘‘Phase 1’’ which learns the category of the
query image. The blue color represents the ‘‘Phase 2’’ which retrieves
similar images from a particular category of images.

Phase 1: Category learning (Green color)

Phase 2: Recommendation (Blue color)

A. ‘‘PHASE 1’’: CATEGORY LEARNING

In Phase 1, the proposed approach models and learns the
class/category of the product based on the image character-
istics. Once the category is selected by Phase 1, Phase 2
retrieves the closely matched similar products from the cor-
responding category.
Figure 3 shows the generic flow of the steps for the image-
based recommendation system. As the proposed approach is
based on the user query image, the approach employs two
phases; one for the category/class learning phase and the other
for a similar image retrieval phase based on the category. The
green color represents the class learning phase. The blue color FIGURE 5. ‘‘Phase 1’’ of the recommendation process as a machine
learning paradigm.
represents the similar image retrieval from the specific class
of images as a recommendation. Phase 2 is further explained
in Figure 6. The model is used to find the class/type of products that are
Figure 4 and Figure 5 show the first phase of the rec- searched or clicked by the user.
ommendation system, which is the expansion of the green For the first ML phase of a product class learning, we use
blocks in Figure 3. The first phase is based on the training and the RF classifier due to its generalization capabilities and
testing paradigm of the ML. For the first phase, the approach increased performance in state of the art. For learning the
takes advantage of the ML for learning the features of the class of the products and for extracting image features, we use
image/product characteristics and generates a learned model. the JPEG coefficients as image features. The RF is further

VOLUME 8, 2020 3311

 F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

integrated into the DL setup for performance analysis and

performance enhancements.
Figure 5 shows the steps of JPEG features being used
by the RF classifier for category learning. From Figure 5,
the available image dataset is used for learning features and
the type of product. The features are extracted based on the
JPEG coefficients. The Coefficients are converted to a feature
vector. This feature vector constitutes the feature set for the
RF classifier. The RF classifier learns the distribution of the
features for different categories of objects. The final model
is stored for finding the corresponding category of the query
image. Phase 1 in Figure 5 is the crucial step of the proposed
recommendation system. The mistake in category prediction
will lead to wrong retrievals in Phase 2. For a query image,
of Figure 3, in Phase 1, the JPEG features are extracted. The
RF classifier uses the trained model to predict the category of
the query image. Once the category is retrieved, the image is
processed by Phase 2.

B. ‘‘PHASE 1’’: DEEP FEATURES INTEGRATION

Figure 6 shows the Deep Learning-based RF (DL-RF)
approach. The structure of the proposed DL-RF for feature
extraction and classification is composed of many layers. The
DL-RF is comprised of 5 Convolution Layers (CL). The CL
is followed by 3 fully connected layers. Each layer uses a
kernel for filtering dimensions. The Kernel coefficients are
incrementally calculated during the training phase. The dot
product is used for the input and weight vectors. Each neuron
is linked to all outputs. Each layer uses the ReLU for expedit-
ing the learning process. The Softmax layer takes input from
the last fully connected layer and generates a probabilistic
distribution of the category of the images. Figure 6 shows
the proposed architecture for enhancing the performance of
the category classification of Phase 1. Features from DL are
obtained from the C7 layer and RF learns the class distribu- FIGURE 6. Random Forest integration into deep learning. The convolution
tion based on these DL features. layers (first five) are followed by two fully connected layers. The random
forest learns and decides the category of the test image.

C. PHASE 2: IMAGE BASED RECOMMENDATION similarity scores are then sorted in the ascending order, and
Figure 7 shows the flow of Phase 2 for recommending similar the top 20 are selected as the possible candidates for the
products that are closely matching the query image. For a recommendations as shown in Figure 7.
query image, the image category is retrieved by Phase 1. The For most of the recommendation systems in the state of the
query image is then searched in the corresponding category art, the similarity index is the last step for recommending sim-
in Phase 2. It retrieves related images based on the similarity ilar products. However, in the proposed approach, we extend
in a particular category. As shown in Figure 7, the category the similarity by introducing image matching steps. Though
images are loaded with the query image. The JPEG features vector matching retrieves similar images, however, the users
are extracted from all the images in a particular category. The are generally concerned about not only item matching but
JPEG features are extracted for the query image as well. This also similar color matching of items. For this reason, and
puts both the category images and the query image in the same for robust matching, we introduce further novel steps. After
vector space. the 20 most similar vectors are retrieved, the images related
The next step is finding the similarity between the feature to these vectors are re-loaded. Then these 20 images are
vector of the query image and the feature vectors of the analyzed by image-based similarity matching. For image-
category images. For similar products selection and vector based similarity matching, we adopt the Structural Similarity
matching, we use the Euclidean distance between all the Index (SSIM) of [28] to include color histograms in the
vectors of the category images to the vector of the query matching process. This is represented as the ‘‘Struct-Hist’’
image. These are represented as the similarity values of the matching process. The SSIM is modified to include the color
query image with all the images in a particular category. The distribution (histogram) as follows:

3312 VOLUME 8, 2020

F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

FIGURE 8. Example images from the dataset.

where A, B, and C are the R, G, B, and N is the total number

of pixels in an image.
We combine Equations 3, 4, 5, and 6 for proposing a
Structural Histogram and represent it as ‘‘Strcut-Hist’’. The
Struct-Hist is the weighted combination of all the parameters
as follows:
Struct−Hist (x, y) = [L(x, y)α .C(x, y)β .Struc(x, y)γ .h] (7)
where L is the Luminance, C is Contrast, and Struc is
the structure parameter. The h represents the histogram and
∝, β, γ are the weighting parameters that are set to 1 for
giving equal importance to all the parameters. The values of
the Struct-Hist are scaled between 0 and 1. A value 1 means
complete matching, 0 implies no match, and values between
0 and 1 mean the similarity and or dissimilarity between two
images. The Struct-Hist combines the structural properties
and the color distribution in the images. It has a number of
benefits. It not only retrieves similar items but weights also
the items with similar colors.
In Figure 7, a total of 20 items are retrieved based on
the Euclidian distance between category image vectors and
FIGURE 7. ‘‘Phase 2’’ of the recommendation process. The ‘‘phase 2’’ the query image vector. The 10 most similarly related prod-
retrieves similar images that of the query image after the category of the ucts out of 20 products are selected by the Struct-Hist
product is identified by the ‘‘Phase 1’’. Struct-Hist is the combination of
structure and color for matching.
approach. With this approach, we believe that the combina-
tion of two steps of finding the 20 similar items based on the
For an image, the luminance (L), contrast (C) and structure Euclidian distance and the most relevant 10 items by Struct-
(Struc) are represented as: Hist always produce very accurate recommendations. The
Struct-Hist based recommendation is compared with the state
2µx µy + c1 of the art approaches in the Experimental section.
L(x, y) = (3)
µ2x + µ2y + c1
V. EXPERIMENTAL EVALUATION
2σx σy + c2
C (x, y) = 2 (4) In this section, we present the dataset, performance evalua-
σx + σy2 + c2 tion of the proposed Phase 1 and Phase 2 of the proposed
σxy + c3 recommendation system. The related parameters and factors
Struc (x, y) = (5)
σx σy + c3 are also analyzed.

where, A. DATASET AND FEATURES

µx the average of x; We use the Amazon product image data. We select a
µy the average of y; dataset of 3.5 million products that consists of 20 categories.
σx2 the variance of x; Figure 4 shows the distribution of the category labels of the
σy2 the variance of y and dataset. We randomly select 100 images for each class. The
c3 = c2 2

dataset is available at [34]. Example images from the dataset
An image histogram is a mass probability function of the are shown in Figure 8.
intensity values in an image. It is employed for color images,
capturing the intensity distribution of the three color chan- B. ‘‘PHASE 1’’ PERFORMANCE ANALYSIS AND
nels. The color histogram is defined as, EVALUATION
For Phase 1 of the proposed approach, we use the RF meta-
hA,B,C (a, b, c) = N .Prob(A = a, B = b, C = c) (6) classifier due to its generalization capabilities and increased

VOLUME 8, 2020 3313

 F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

performance in state of the art [29]–[31]. The features learned

by the RF are based on JPEG coefficients. There are mul-
tiple reasons for choosing the JPEG Coefficients as feature
vectors. The JPEG coefficients are thoroughly researched
for extracting important features from images for compress-
ing and reducing their overall size. Thus the overall image
quality is maintained while sufficient image size reduction
is achieved. As such, these coefficients represent the critical
information in the image. From the implementation point
of view, the JPEG coefficients are computationally feasible
and efficient due to the DCT transformation. The transfor-
mation is regarded as a set of basis functions; thus they are
FIGURE 9. Performance evaluation of Phase 1 based on Precision. The
efficiently precomputed and stored for the re-construction of X-axis shows the % Precision values. ‘‘Deep learning’’ represents the
compressed images. Also, the choice of the JPEG coefficients approach where features are extracted by JPEG and classified by the
Deep learning. ‘‘Deep learning with RF’’ is the approach where the
as feature vectors is due to the pre-comparative analysis in our features are extracted by the Deep learning and then those features are
work. In the preliminary experiments, we compared the fea- learned by the random forest.
tures of HOG, Color Layout, and auto-correlogram features.
We obtained the highest performance for the JPEG features
in all cases. Also, compared to the HOG, Color Layout, and F-measure takes both the precision and recall into consid-
auto-correlogram features, the JPEG feature calculation is eration as follows:
much faster. Therefore, based on all these parameters, we use Precision ∗ Recall
JPEG features. F − measure = 2 ∗ (10)
The performance of a particular classifier depends on Precision + Recall
the nature of data and features. For performance analysis The RF approach of Phase 1 is compared with the state of
of Phase 1, we focus on the 10-folds cross-validation as a the art machine learning models. Following are the main
training and testing paradigm for the validation of the class comparative approaches:
prediction. The 10-folds cross-validation not only reliably Logistic Regression: is the baseline algorithm for classifi-
learns and tests the performance of the classifier, but is also cation and regression. It is used for multi-class classification
a standard approach for classifier comparison in state of the problems [18].
art. The 10-folds cross-validation uses 90% data for training Naïve Bayes: is used in many classifications and regres-
and 10% data for testing. For 90% of training data, the model sion problems and is based on the independence property of
is created and tested on the 10% testing data. The testing the variables in the training data [19].
data is un-seen data for creating the model. Therefore, this SVM: takes data to a higher dimension and uses hyper-
process represents the real-world scenario for new queries planes to separate different classes. SVM has been used in
to the recommendation system of Phase 1. The prediction many machine learning-based image classification tasks [20].
performance from the first round is stored. The process is DL: DL algorithm is the state-of-the-art approach. DL
repeated 10 times, and the average performance is calculated can be used as a simple classifier, for which the non-Deep
for all the 10 rounds. It thus removes the bias of results, and methods extract data features and DL only learns the class
the performance can be generalized to practical applications. separation. This approach is not as robust as the one where
For the evaluation of Phase 1, we use the Precision, Recall, DL is used to extract features. DL can also be used in the sec-
and the F-measure. The precision and recall are favored over ond approach, for which the features are learned and also
the accuracy when the classes are not balanced. Since we have classified by the DL algorithm itself. The DL in this setting
unbalanced classes in the dataset, we use these parameters is used extensively in image classification and regression
for evaluation. The F-measure takes both the Precision and problems [21]. In the comparative experiments, when we use
Recall for calculation and is a reliable parameter for similar DL, we mean the usage of DL for classifying JPEG features.
applications. The Precision is calculated as: Deep Learning with RF: In this setup, DL is integrated
into the RF setup. The DL learns the features using the
TP CNN model and classifies them by the RF classifier. The
Precision = (8) RF integration into the DL setup is shown Figure 6. The 5
TP + FP
convolution layers are followed by two fully connected lay-
where TP is True Positive and FP is False Positive. ers. After the DL features are obtained from the C7 layer,
the features are then modeled by the RF classifier.
TP Figure 9 shows the performance evaluation of Phase
Recall = (9)
TP + FN 1 based on Precision. In Figure 9, the ‘‘Deep Learning’’
represents the approach where features are extracted by the
FN is False Negative. JPEG approach and only classified by the DL. The ‘‘Deep

3314 VOLUME 8, 2020

F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

FIGURE 10. Performance evaluation of Phase 1 based on Recall. The FIGURE 11. Phase 1 F-measure evaluation. The X-axis shows the %
X-axis shows the % recall values. ‘‘Deep learning’’ represents the F-measure values. ‘‘Deep learning’’ represents the approach where
approach where features are extracted by JPEG and classified by the features are extracted by JPEG and classified by the deep learning. ‘‘Deep
Deep learning. ‘‘Deep learning with RF’’ is the approach where the and learning with RF’’ is the approach where the features are extracted by the
then those features are learned by the random forest. Deep learning and then those features are learned by the random forest.

learning with RF’’ in Figure 9 depicts the proposed method in step, it is still labeled by humans, and thus the results are
which the features are extracted by the DL and then classified subjective.
and learned by the RF. From Figure 9, the Logistic Regression Therefore, we adopted the Autocorrelogram vector-based
obtains a very low precision of 38.2%. The NaïveBayes has Euclidian distance for evaluation of Phase 2. We use the
slightly improved precision of 43%, The SVM with preci- Autocorrelograms of [33] for feature vectors in Phase 2 to
sion of 50.4% is better than the Logistic Regression, Naive remove the biases that can be induced by the JPEG feature
Bayes, and the DL (used as the Neural Network for learning vectors used in the retrieval of Top 20 images. We could also
class distribution only). The RF gets an increased Precision use the JPEG vectors, but this will bias the results towards
of 74.9%. The Precision of 74.9% is further enhanced to our approach because the training vectors are the JPEG
85% by integrating the RF with the DL paradigm as shown vectors. Secondly, the Autocorrelograms has shown good
in Figure 9. performance for retrievability and generalization of image-
Figure 10 shows the Recall evaluation of Phase 1. In based retrieval [33]. Third, the Autocorrelogram of the image
Figure 10, the Logistic Regression gets a very low Recall captures the spatial correlation between similar intensities in
of 37%, The NaïveBayes has slightly improved Recall, The the corresponding images.
SVM Recall is better than the Logistic Regression, Naive For the comparative evaluation of Phase 2, we choose the
Bayes, and the DL. The RF gets an increased Recall of 75%. K-Nearest Neighbor (KNN) and Search-based approaches.
The Recall of 75% is further enhanced to 83% by integrating As such, we conducted experiments involving 100 query
the RF with the DL approach as shown in Figure 10. images. These images are randomly selected from the dataset.
Figure 11 shows the performance evaluation of Phase 1 in Every image is used as the query image for Phase 2 for the
terms of F-measure. In Figure 11, for the DL, features are retrieval of ten similar images.
extracted by the JPEG approach and only classified by the For evaluation, the Autocorrelograms of the ten similar
DL. The ‘‘Deep learning with RF’’ in Figure 11 represents images and the query images are calculated. Then each
the proposed approach where the features are extracted by the retrieved image (vector) is compared to the query image by
DL and then learned by the RF. From Figure 11, the Logistic three approaches. These comparative approaches are:
Regression gets a very low F-measure of 37.6%. The Naïve-
Bayes has slightly improved F-measure of 41%, The SVM 1. Subjective similarity (by 3 users)
with F-measure of 47% is better than the Logistic Regres- 2. Euclidian distance-based similarity
sion, Naive Bayes, and the DL. The RF gets an increased 3. Cosine similarity
F-measure of 75%. The F-measure of 75% of RF is further For subject similarity, the vectors are converted to their
enhanced to 84% by integrating the RF with the DL approach, original images for visualization purposes and presented to
as shown in Figure 11. users. Three students (users) perform subjective evaluation.
The images are retrieved for the 100 test queries using the
C. ‘‘PHASE 2’’ EVALUATION Struct-Hist, the KNN, and the Search-based approach. These
The evaluation of Phase 2 of the recommendation system is images are then arranged with retrieved labels being removed.
not straight forward as that of Phase 1. This is due to several The labels are removed so that the user has no clue of the
reasons. First, the recommendation is a subjective process. algorithm used for retrieval for an un-bias scoring. Each
Secondly, there is no baseline for comparison. Third, even user has to give a score between 0 and 10 to the retrieval
if the data is labeled for Phase 2 of the recommendation performance of the particular approach for the 10 retrieved

VOLUME 8, 2020 3315

 F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

FIGURE 12. Subjective evaluation. The X-axis shows the average FIGURE 14. Cosine similarity. Evaluation of the recommendation phase
similarity score of test images. ‘‘Phase 2’’ Struct-Hist step of the proposed approach by the Cosine
similarity. The X-axis shows the average similarity score of test images.

FIGURE 13. Evaluation of the recommendation phase ‘‘Phase 2’’

Struct-Hist step of the proposed approach by the Euclidian similarity. The
X-axis shows the average similarity score of test images.
FIGURE 15. Example retrievals.

images. The scores are then averaged and scaled between

0 and 1. Figure 13 shows the statistics of an average similarity
Figure 12 shows the result of the comparative subjec- of 100 query images. The higher the similarity value,
tive evaluation by three users. Figure 12 shows the average the better the retrieval performance. Figure 13 shows the
subjective similarity of 0.93 for the proposed ‘‘Struct-Hist’’ average similarity of 0.98 for the proposed ‘‘Struct-Hist’’
approach. For the KNN, the average similarity is also 0.93. approach. For the KNN, the average similarity is 0.96.
Search-based retrieval has an average similarity of 0.929. Search-based retrieval has an average similarity of 0.95. The
An interesting result is obtained in the subject evaluation. Euclidian similarity statistics show a good overall perfor-
It is seen that almost all three approaches have very similar mance for the proposed Struct-Hist approach.
similarity scores. This can be due to the fact that human The Cosine similarity between two vectors is calculated as:
ignores some of the details of the images and focuses more Pn
A.B Ai Bi
on global image features. Therefore, for a particular user, it is Cos_sim = = qP i=1qP , (12)
found that the difference between an average similarity for kAk kBk n 2 n 2
i=1 Ai i=1 Bi
a particular approach is not that significant. This result also
shows that as long as an image is slightly related to the query where Ai and Bi are the components of image vectors
image/product, the human generally considers it a complete A and B.
match. The Cosine similarity is scaled between 0 and 1, where
The Euclidian similarity is based on the vector (image) 0 represents 0% similarity, and 1 means 100% similarity.
distance as follows: All values between 0 and 1 indicate the corresponding per-
r
Xn centage of similarity. Figure 14 shows the results of the
Euc_Sim = (qi − pi )2 , (11) comparative evaluation using the Cosine similarity for the
i=1
three approaches. Figure 14 depicts the average similarity of
where p and q are two vectors, the Euclidian distance is then 0.965 for the proposed ‘‘Struct-Hist’’ approach. For the KNN,
used as the similarity and scaled between 0 and 1, where the average similarity is 0.959. Search-based retrieval has an
0 represents 0% similarity, and 1 means 100% similarity. All average similarity of 0.95. The Cosine similarity of Struct-
the other values between 0 and 1 indicate the corresponding Hist is higher compared to the other two approaches.
percentage of similarity. As the proposed approach consists of two steps. First
Figure 13 shows the result of the comparative evalua- retrieving 20 images, and then selecting 10 images by the pro-
tion using the Euclidian similarity for the three approaches. posed Struct-Hist approach. We believe that the combination

3316 VOLUME 8, 2020

F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

TABLE 1. Parameters and time complexity. image recommendation. The article contributes not only to
the recommendation based systems but the algorithm pre-
sented in this article can be used for generic computer vision
problems. In the future, we aim to merge non-image informa-
tion with the images and use the Recurrent Neural Network
(RNN) architecture for the recommendation process.

REFERENCES
[1] I. Kanellopoulos and G. G. Wilkinson, ‘‘Strategies and best practice for
neural network image classification,’’ Int. J. Remote Sens., vol. 18, no. 4,
pp. 711–725, 1997.
[2] R. He and J. McAuley, ‘‘Ups and downs: Modeling the visual evolution
of fashion trends with one-class collaborative filtering,’’ in Proc. 25th Int.
Conf. World Wide Web, 2016, pp. 507–517.
[3] J. McAuley, C. Targett, Q. Shi, and A. van den Hengel, ‘‘Image-based
recommendations on styles and substitutes,’’ in Proc. 38th Int. ACM SIGIR
Conf. Res. Develop. Inf. Retr., 2015, pp. 43–52.
[4] V. Jagadeesh, R. Piramuthu, A. Bhardwaj, W. Di, and N. Sundaresan,
‘‘Large Scale Visual Recommendations From Street Fashion Images,’’
of two steps in the proposed approach always produces accu- in Proc. 20th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining,
rate recommendations. The Struct-Hist approach thus outper- pp. 1925–1934, 2014.
[5] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ‘‘Imagenet classification
forms the others and shows its efficacy for the image-based with deep convolutional neural networks,’’ Commun. ACM, vol. 60, no. 6,
product recommendations. Figure 15 shows some examples pp. 84–90, May 2017.
of retrievals. [6] K. Simonyan and A. Zisserman, ‘‘Very deep convolutional networks for
large-scale image recognition,’’ in Proc. 3rd Int. Conf. Learn. Represent.
(ICLR), May 2015, pp. 1–14.
D. TIME COMPLEXITY [7] G. Wang, D. Hoiem, and D. Forsyth, ‘‘Learning image similarity from
flickr groups using fast kernel machines,’’ IEEE Trans. Pattern Anal. Mach.
Table 1 shows the time taken by different modules. The time
Intell., vol. 34, no. 11, pp. 2177–2188, Nov. 2012.
is represented in seconds and is the average of several runs. [8] G. W. Taylor, I. Spiro, C. Bregler, and R. Fergus, ‘‘Learning invariance
The experiments are conducted using the Core I7, running the through imitation,’’ in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern
Titan Nvidia GPU. Since there are many routines tested in this Recognit., Jun. 2011, pp. 2729–2736.
[9] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen,
paper, we report the time for different modules. As in Table 1, and Y. Wu, ‘‘Learning fine-grained image similarity with deep ranking,’’ in
the average time taken for calculating and learning the JPEG Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., Jun. 2014,
features by RF is 35 seconds. The time of finding the category pp. 1386–1393.
[10] A. Dosovitskiy and T. Brox, ‘‘Generating images with perceptual similarity
of the query image by the JPEG and RF is 0.03 seconds. The metrics based on deep networks,’’ in Proc. 30th Int. Conf. Neural Inf.
time for calculating and learning the Deep features by RF is Process. Syst., 2016, pp. 658–666.
1000 seconds. Time to find the category of the query image [11] M. Tan, S. Yuan, and Y. Su, ‘‘Content-based similar document image
retrieval using fusion of CNN features,’’ Commun. Comput. Inf. Sci.,
by the Deep features and RF is 0.3 seconds. The average vol. 819, pp. 260–270, Mar. 2018.
time taken to find the top 20 related items in ‘‘Phase 2’’ is [12] S. Bell and K. Bala, ‘‘Learning visual similarity for product design with
1 second. Time to retrieve the 10 out of 20 associated items convolutional neural networks,’’ ACM Trans. Graph., vol. 34, no. 4, p. 98,
in ‘‘Phase 2’’ is 0.003 seconds. 2015.
[13] T. Deselaers and V. Ferrari, ‘‘Visual and semantic similarity in ima-
genet,’’ in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.,
VI. CONCLUSION Jun. 2011, pp. 1777–1784.
[14] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, ‘‘From image descrip-
We presented an image-based product recommendation sys- tions to visual denotations: New similarity metrics for semantic infer-
tem based on two phases. In Phase 1, the proposed approach ence over event descriptions,’’ Trans. Assoc. Comput. Linguistics, vol. 2,
learns the class/type of the product. In Phase 2, the proposed pp. 67–78, Feb. 2014.
recommendation system retrieves closely matched similar [15] J. Yang, J. Fan, D. Hubball, Y. Gao, H. Luo, W. Ribarsky, and M. Ward,
‘‘Semantic image browser: Bridging information visualization with auto-
products. From the ML phase of a product’s class learn- mated intelligent image analysis,’’ in Proc. IEEE Symp. Vis. Analytics Sci.
ing, we used the RF classifier. For feature extraction from Technol., Oct./Nov. 2006, pp. 191–198.
images, we used the JPEG coefficients as image features. [16] C. Cortes and V. Vapnik, ‘‘Support-vector networks,’’ Mach. Learn.,
vol. 20, no. 3, pp. 273–297, 1995.
In the evaluation of Phase 1, the proposed model generates [17] L. Breiman, ‘‘Random forests,’’ Mach. Learn., vol. 45, no. 1, pp. 5–32,
a 75% accurate model. For performance enhancements, the 2001.
RF model is further integrated into the DL setup and achieves [18] L. J. Li, H. Su, E. P. Xing, and L. Fei-Fei, ‘‘Object bank: A high-level image
representation for scene classification & semantic feature sparsification,’’
84% correct predictions. In Phase 2, we believe that the com- in Proc. Adv. Neural Inf. Process. Syst., 2010, pp. 1–9.
bination of two steps in the proposed approach of finding the [19] S. McCann and D. G. Lowe, ‘‘Local Naive Bayes Nearest Neighbor for
20 similar items based on the Euclidian distance and 10 most image classification,’’ in Proc. IEEE Comput. Soc. Conf. Comput. Vis.
similar out of 20 by the Struct-Hist approach produces very Pattern Recognit., Jun. 2012, pp. 3650–3656.
[20] Y. Tarabalka, M. Fauvel, J. Chanussot, and J. A. Benediktsson, ‘‘SVM- and
accurate recommendations. The Struct-Hist approach thus MRF-based method for accurate classification of hyperspectral images,’’
outperforms the other methods and shows its efficacy for the IEEE Geosci. Remote Sens. Lett., vol. 7, no. 4, pp. 736–740, Oct. 2010.

VOLUME 8, 2020 3317

 F. Ullah et al.: Image-Based Service Recommendation System: A JPEG-Coefficient RFs Approach

[21] Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu, ‘‘Deep learning-based BOFENG ZHANG received the B.S., M.S., and
classification of hyperspectral data,’’ IEEE J. Sel. Topics Appl. Earth Ph.D. degrees from the Northwestern Polytech-
Observ. Remote Sens., vol. 7, no. 6, pp. 2094–2107, Jun. 2014. nical University of Technology, Xi’an, China,
[22] Nielsen. (2015). Green Generation: Millennials Say Sustainability is in 1991, 1994, and 1997, respectively. From
a Shopping Priority. The Nielsen Global Survey of Corporate Social 1997 to 1999, he worked in Zhejiang University
Responsibility and Sustainability. Accessed: May 3, 2019. [Online]. as a Postdoctoral Researcher and was promoted to
Available: https://www.nielsen.com/us/en/insights/article/2015/green- an Associate Professor. Since 1999, he has been
generation-millennials-say-sustainability-is-a-shopping-priority/
on the faculty of the School of Computer Engi-
[23] L. Liu, ‘‘Contextual topic model based image recommendation system,’’
neering and Science, Shanghai University. From
in Proc. IEEE/WIC/ACM Int. Jt. Conf. Web Intell. Intell. Agent Technol.
(WI-IAT), Dec. 2015, pp. 239–240. August 2006 to August 2007, he worked as a Vis-
[24] I. Memon, L. Chen, A. Majid, M. Lv, I. Hussain, and G. Chen, ‘‘Travel iting Professor at the University of Aizu, Japan. From September 2013 to
recommendation using Geo-tagged photos in social media for tourist,’’ September 2014, he worked as a Visiting Professor at Purdue University
Wireless Pers. Commun., vol. 80, no. 4, pp. 1347–1362, 2015. Calumet, USA. He is currently the Vice Dean of the School of Computer
[25] Y. Sun, H. Fan, M. Bakillah, and A. Zipf, ‘‘Road-based travel recommen- Engineering and Science, Shanghai University. He has published three
dation using geo-tagged images,’’ Comput. Environ. Urban Syst., vol. 53, books, about 150 research articles in national and international journals and
pp. 110–122, Sep. 2015. major conference proceedings. His current research interests include intel-
[26] L. Yu, F. Han, S. Huang, and Y. Luo, ‘‘A content-based goods image recom- ligent information processing, data science and technology, and intelligent
mendation system,’’ Multimed. Tools Appl., vol. 77, no. 4, pp. 4155–4169, human–computer interaction. He serves as a steering committee member,
2018. a Workshop Chair, and a program committee member of several important
[27] S. C. Guntuku, S. Roy, and L. Weisi, ‘‘Personality modeling based international conferences.
image recommendation,’’ in Proc. Int. Conf. Multimedia Modeling, 2015,
pp. 171–182.
[28] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, ‘‘Image quality
assessment: From error visibility to structural similarity,’’ IEEE Trans.
Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[29] R. Khan, A. Hanbury, J. Stöttinger, and A. Bais, ‘‘Color based skin classi-
fication,’’ Pattern Recognit. Lett., vol. 33, pp. 157–163, Jan. 2012.
[30] R. Khan, A. Hanbury, and J. Stoettinger, ‘‘Skin detection: A random
forest approach,’’ in Proc. Int. Conf. Image Process. (ICIP), Sep. 2010,
pp. 4613–4616.
[31] B. Xu, Y. Ye, and L. Nie, ‘‘An improved random forest classifier for
image classification,’’ in Proc. IEEE Int. Conf. Inf. Automat., Jun. 2012,
pp. 795–800.
[32] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ‘‘ImageNet classification
with deep convolutional neural networks,’’ in Proc. Adv. Neural Inf. Pro-
cess. Syst., 2012.
[33] J. Huang, S. R. Kumar, M. Mitra, W.-J. Zhu, and R. Zabih, ‘‘Image indexing
using color correlograms,’’ in Proc. IEEE Comput. Soc. Conf. Comput. Vis.
Pattern Recognit., Jun. 1997, pp. 762–768.
[34] Amazon Dataset. Accessed: Sep. 3, 2019. [Online]. Available:
https://drive.google.com/drive/folders/1eaCPVw-j9ORkVHA4JY-
n0QFXkHmilukm

REHAN ULLAH KHAN received the B.Sc.

FARHAN ULLAH received the B.Sc. degree in and M.Sc. degrees in information systems from
computer science from the University of Peshawar the University of Engineering and Technology,
and the M.C.S. degree from the Sarhad University Peshawar, in 2004 and 2006, respectively, and
of Information Science and Technology, Peshawar. the Ph.D. degree from the Vienna University of
He is currently pursuing the Ph.D. degree in com- Technology, Austria, in 2011. He is currently an
puter science with Shanghai University, Shanghai, Assistant Professor with the IT Department, CoC,
China. His research interests include machine Qassim University, KSA. His current research
learning, recommendation systems, and computer interests include segmentation, machine learning,
vision. recognition, and image-based object recognition.

3318 VOLUME 8, 2020