Research Article

Research on Cloud Computing Resources Provisioning

Based on Reinforcement Learning

1. Introduction
As one of the core issues for cloud computing, resource
The concept of cloud computing vividly reflects the charac- management aims to shield the underlying resource hetero-
teristics of information service in Internet age; meanwhile geneity and complexity by adopting virtualization technol-
the pursuit of the vision of cloud computing also brings ogy, which makes the massive distributed resources form
new challenges to information technology. Acting as a a unified giant resource pool. Therefore, it can guarantee
significant application research, the data center is pushing the efficient resource provision and use by implementing
a series of technology innovations to perform the key resources management methods and techniques rationally.
features of cloud computing, such as on-demand service, Therefore, how to achieve effective management of cloud
elasticity of extension, and massive data storage. The data computing resources is faced with a number of new chal-
center widely adopts virtualization technology to achieve lenges, which are mainly shown in three types of
the uncoupled mode of physical resource and application. imbalance.
Applications use Virtual Machine (VM) as a package unit
to share various physical resources with others. Hence the (i) First, Imbalance in the Needs of Applications. Cloud
resource schedule entities are represented by fine-grain com- puting application contains various behaviors of
VMs instead of coarse grain service machines. workload, from the control-intensive applications (such
Virtualization technology provides convenience for the data as search, sort, and analysis) to the data-intensive ones
center, but the VMs resource provi- sioning brings more (image processing, simulation, modeling, data mining,
challenges to the efficient management of data center etc.). In addition, it also includes the computationally
infrastructure. intensive applications (itera- tive method, numerical
method, financial modeling, etc.). The throughput of
various applications depends heavily on
2 Mathematical Problems in Engineering

the VM resource provisioning, while not even any

apparently lagging behind, affecting the user’s experience
configura- tion can make all types of workloads run with and even being unable to meet some of the user requests at
optimal effi- ciency. Furthermore, most applications are times. In conclusion, the data center must solve the
featured by mul- tiple types of workload. For example, problems such as resource multiplexing, correlation, and
control-intensive appli- cations require more CPU dynamic management. High-efficiency dynamic
resources for branch prediction, while data-intensive ones management for vir- tualization resources is the core issue
require more memory resources to avoid the frequent of the optimal resource scheduler and also the key of
operation of reading and writing. The multitenant how various resource ser- vice systems eventually provide
environment of cloud computing allows hetero- geneous appropriate and satisfying
applications to share data center resources pool, and the resources for users.
demand for resources for each application is diverse, thus Focusing on the accurate scaled cloud computing envi-
resulting in difficulty measuring the server-loading ronment and efficient resources allocation under Service
efficiency. Even for the different resources of the same Level Agreement (SLA) and user cost constraints, we intro-
server, it is prone to cause an imbalance, affecting the duce two concepts, Segmentation SLA (SSLA) and
resource use efficiency. In cloud computing scenario, while Utilization Unit Time Cost (UUTC), and then propose an
simply the demand for fore- casting and the rational optimization resource provisioning scheme based on
purchase cannot solve this problem, a plausible resource reinforcement learn- ing (RL) and queuing theory (QT) in
provisioning scheme needs to be put for- ward to further this paper.
solve the new contradiction between hetero- geneous This research work has the following theoretical and
practical contributions.
application and the unified resources-sharing pool.
(i) A novel dynamic resources provisioning scheme
(ii) Second, Imbalance in the Application Time. In reality, based on QT and RL is proposed.
the server utilization of data center reaches merely 5% to
(ii) Two concepts are introduced to evaluate the perfor-
20%, while the peak workloads of many services are 2–10
mance of various resources provisioning scheme.
times higher than the average. In addition to the different
service loads in various periods of a day, most of the (iii) To demonstrate our method, we apply our method
services vary in load demands according to the seasonal or to the simulation and real resources provisioning for
other periodic changes (e.g., peak in December before cloud computing platform.
Christmas sales and in photo processing sites after holiday); (iv) The experiment results demonstrate that our devel-
meanwhile some unexpected events (such as news) lead to oped method can make accurate provisions at
changes. Few users deploy less resource than the peak numer- ous arrival rates of users’ job and avoid SLA
demands, which is prone to waste the resources at the conflicts.
nonpeak time. Asa result, the stronger the load is The remainder of this paper is organized as follows.
fluctuating, the more resources the users are wasting. In the Section 2 reviews the related work of resources
cloud computing environment, this issue cannot be handled provisioning in cloud computing environment. Section 3
by static configuration mode. At the same time, the VM in explores the construction of cloud computing platform and
the cloud is characterized by isolation performance, but detailed anal- ysis on the implementation of user job in
mutual interference resulting from the resource competition proposed system model. Based on the cloud models, we
between VMs cannot be avoided in the actual operation of introduce two new concepts: SSLA and UUTC,
the system process, affecting the performance of the whole respectively. In Section 4, we design a resources allocation
cloud computing system. scheme based on reinforcement learning; then according to
the shortage of basic 𝑄 learning, we propose an improved
(iii) Thirdly, Imbalance in the Distribution of Applications. 𝑄 learning scheme to enhance the algorithm performance.
For load balancing, the node servers of the load equalizers The experiment results are presented in Section 5, and,
are not fixed physical machines but are VMs of the cloud, finally, we reach the conclusions and skeleton of our future
which require the load balancer to be equipped with the work in Section 6.
ability of dynamically adjusting the server cluster to the
current user access, so as to avoid the resource waste and 2. Related Work
the situation that the current resources cannot meet the
user’s requests [1]. To some extent, numerous academic Dynamic resource management in cloud computing
researches have been conducted on the trend prediction of environ- ment refers to the process of dynamic optimization
the load and elastic assignment of resources; however, alloca- tion, organization, coordination, and control of
there are still some disadvantages. Having conducted a resources. It should not only support task scheduling in
comprehensive study of academia and industry current interorganiza- tional or management domain, real-time
status, we find that the research in this field has the resources monitor- ing, and job execution, but also maintain
following problems: first, it lacks flexibility. Apart from the self-manage- ment of the local sites, providing the
not considering the dynamic deployment of resources from corresponding Quality of Service (QoS) support. It is an
a service-oriented perspective, some characteristics are not advanced form of resource management and also the core
fully reflected, such as the elasticity of cloud computing component of resource man- agement system in cloud
and the characteristic that resources are adjusted to the computing environment, so as to shield the heterogeneity
user’s needs. Second, it does not support the trend and complexity of the underlying resources, to manage the
prediction, and the resource allocations are
Mathematical Problems in Engineering 3
distributed massive resources in the cloud computing, to
control the resources effectively, to
improve resource utilization, and to provide the reasonable
PM to solve the problem. In [12], the authors focused on
distribution of resources operation for cloud computing,
cloud backup optimization problem, employing two
thus to balance the load.
decision parameters and finite-source queueing theory to
Since the concept of cloud computing was proposed,
maintain the regulated service quality of the cloud platform.
resource scheduling, especially dynamic resources
Distinguished from prior works, we establish a model
provision- ing, has been one of the most important research
of cloud computing system to strengthen the learning as
compo- nents. Related works of resource provisioning are
an optimization tool for dynamic resources provisioning in
mainly from different perspectives to construct a cloud
data center based on the queueing theory in this paper. To
computing system model of queueing theory, aiming to
the best of the authors’ knowledge, not many related papers
attain universal results. However, being affected by factors
have appeared in the literature concerning dynamic
such as the het- erogeneity of the built platforms, the
resources provisioning.
incompatible interface, and the disparity of the underlying
physical resources, various research results demonstrate
discrepancies, difficult to make a comparable analysis 3. System Model
among each other.
In [2], dealing with the combined issues of power and 3.1. Cloud Computing Platform Framework. In
per- formance management in cloud data centers, the this section, we will introduce the framework of the
authors pro- posed a dynamic resource management cloud computing platform used in this study, as
scheme by leveraging both of the techniques such as depicted in Figure 1. The details of system organization
dynamic voltage/frequency scaling and server structure and the functions of various parts are
consolidation, thus to achieve energy efficiency and desired described as follows.
application-level performance. The novelty of the proposed
scheme was its integration with timing analysis, queuing Users Interface (UI). The main function of users interface
theory, integer programming, and control theory (UI) is to receive the user’s requests, allocating them to
techniques. In [3], despite the varying event arrival rates, a the corresponding VM cluster on the basis of provisioning
queuing theory based approach was pursued to achieve algorithm and then receiving the execution results and
specified response time target; by drawing the neces- sary returning them to the user.
computing resources from a cloud, a distinct query engine Users Job Queue (UJQ). Cloud computing platform
was modeled as an atomic unit to predict response times. includes two user-job queues. After they are submitted
Several similar units hosted on a single node were modeled via the user interface to the cloud computing platform,
as a multiple class M/G/1 queuing system and the response the jobs enter queue 1 in turn and wait to be scheduled.
times were deemed to meet specified targets although being When they are executed, they enter the user queue 2 in
subject to varying event arrival rates over time. Correlation turn, waiting to be sent back to the end user by the
work was also extended to multimedia cloud and large web transmitter.
server clusters [4, 5]. In [4], concentrat- ing on resource
allocation problems in multimedia cloud, the authors Users Job Scheduling (UJS). By employing the job
employed optimization methods and queueing the- ory; scheduling policy, the user’s requests in queue 1 are
theoretical analysis and computer simulation demon- strate scheduled to the corresponding VM clusters.
the resource cost minimization problem and the response
time minimization problem, respectively. In [6], the authors Virtual Machine Cluster (VMC). A plurality of VMs to
proposed a dynamic resource allocation scheme to resist perform the same operation type forms a VMC. Each VM
distributed denial of service (DDoS) attacks against of a VMC is especially designed for a certain type of
individual cloud customers. This paper was an early work operation to enable the highest operating efficiency. Each
that discussed mitigating DDoS attacks using resource VMC is equipped with a Virtual Machine Cluster Agent
allocation scheme for individual cloud customers. In [7, 8], (VMCA), responsible for the VM instance generation,
the authors proposed an embedded Markov chain analytical management, and cancellation. For example, when a user
model to estimate cloud performance. In [9], the authors job arrival rate increases, the VMs within the current VMC
focused on SLA-aware service deployment optimization cannot meet QoS or SLA, so VMCA needs to increase the
problem, the designed E3-R which is a multiobjective number of the VMs within the VMC to improve its
genetic algo- rithm, to seek individuals and exhibit in cloud throughput; otherwise, some of the VMs in the VMC need
computing environments. In order to meet both to be cancelled so as to reduce energy consumption.
requirements, E3-R employed two different fitness
VM. Responsible for specific job execution, VM removes
functions for different kinds of individuals. In [10], the
a user job from the queue for execution, transmits the
authors focused on the bottleneck of network I/O and the
execution results to the platform interface, and extracts the
aggregation on the packets delay and proposed a
next user request. Each VM is equipped with a
mechanism based on packet aggregation to achieve the best
Performance Monitor Agent (PMA) and a Resource
tradeoff between the throughput and packets delay. In [11],
Management Agent (RMA). PMA is responsible for the
the authors investigated the elastic resource provisioning
performance indicators to monitor the entire VM, including
problem under the burstiness of incoming requests and
response time, throughout, and resource utility. RMA is
energy consumption, employed the ON-OFF Markov chain
responsible for the VM resource management, mainly
and queueing theory to describe burstiness, and proposed a
including the dynamic scheduling of CPU, memory,
VM consolidation mechanism for each
bandwidth, and data center resources.
 Users interface

User queue
1

User job
scheduling

VM1 VM1 VM1 VM1

PMA

PMA

PMA

PMA
VM2 VM2 VM2 VM2

VM3 VM3 VM3 ··· VM3

RMA
RMA

RMA
RMA

. . . .

VMn VMn VMn VMn

VMCA VMCA VMCA VMCA

VM cluster 1 VM cluster 2 VM cluster 3 VM cluster n

User queue
2

User job
transmit

FIgURE 1: Architecture of cloud computing framework.

Users Job Transmit (UJT). The execution results of queue where 𝜌𝑖𝑗 = 𝜆 𝑖𝑗 /𝜇𝑖𝑗 and the Probability Density Function
2 are transmitted to the corresponding user in light of the (PDF) of JQT is
transmission strategy.

3.2. Job Response Time (JRT). According to the system

model 𝑓JQT (𝑡) = (1 − 𝜌 ) 𝛿(𝑡) +
𝜇𝜌 (1 − 𝜌 ) 𝑒−𝜇𝑖𝑗 (1−𝜌𝑖𝑗 )𝑡
shown in Figure 1 and the job execution phase in cloud 𝑖𝑗 𝑖𝑗 𝑖𝑗
(2)
computing environment, Job Response Time (JRT) hinges (𝑡 ≥ 0)
on Job Queueing Time (JQT), Job Execution Time (JET), .
and Job Transfer Time (JTT). In other words, JRT is made
up of JQT, JET, and JTT.
In light of the classic queueing theory, given the fact Similarly, the response time of JET and JTT [14, 15]
that the job arrival rate of the computing platform is 𝜆, the is
arrival
rate of the 𝑗th VM in the cluster 𝑖 is 𝜆 𝑖𝑗 , and its service rate is
𝜇𝑖𝑗 . Thus, the allocated average queueing time JQT can be
described as [13] 1
JET = ,
(3)
𝜇𝑖𝑗
𝜌𝑖𝑗
𝐷 /𝐵
JQT = , (1) JTT = 𝑖𝑗 𝑖𝑗 ,
𝜇𝑖𝑗 (1 − 𝜌𝑖𝑗 ) 1 − 𝜆 𝑖𝑗 𝐷𝑖𝑗 /𝐵𝑖𝑗
where 𝐷𝑖𝑗 is the result size of user job and 𝐵𝑖𝑗 is the
provisioned bandwidth resources of the user job. And the TABLE 1: Amazon EC2 pricing for standard on-demand instances.
PDF of JET and JTT is
Instance type Linux (per hour) Windows (per hour)
−𝜇𝑖𝑗 𝑡 Small (default) $0.060 $0.115
𝑓JET (𝑡) = 𝜇𝑗𝑖 𝑒 (𝑡 ≥ 0) , (4)
Medium $0.120 $0.230
Large $0.240 $0.460
𝐵𝑖𝑗 −(𝐵 /𝐷 )𝑡
𝑓JTT (𝑡) = 𝑒 𝑖𝑗 𝑖𝑗 (𝑡 ≥0 ) , (5) Extra large $0.480 $0.920
𝐷𝑖𝑗
crestLin
respectively.
es
Thus the total response time in the VMC can be given as

𝑇tot = JQT + JET + JTT

crestMa
𝜌
𝐷 / (6) tch
𝐵
1
𝑖𝑗 𝑖𝑗 𝑖𝑗
= + .
𝜇𝑖𝑗 (1 − 𝜌𝑖𝑗 ) 𝜇 𝑖𝑗 1 − 𝜆 𝑖𝑗 𝐷𝑖𝑗 /𝐵𝑖𝑗 Yasmina Baladin
PFMatchICP
3.3. Segment SLA. The response time, which is the perfor-
mance indicator of the cloud computing platform, is con-
strained by QoS or SLA. In this study, we divide SLA into
PFRegister
the varying phases at which the user job is to be executed, Accurac
so as to provision the resources accurately in the cloud y
computing environment. When the job is run, the resources evaluati
on
can be provisioned according to the different stages by the
resource provisioning scheme, thus to enable each phase to FIGURe 2: Medical image analysis application.
be con- strained by SLA, as indicated in the following
representation:

JQT ≤ SLAJQT, terms of resource utilization rate and improves the optimized
JET ≤ (7) function.
As for a user’s job, the issue on resource provisioning
SLAJET, JTT
optimization in cloud computing platform can be denoted as
≤ SLAJTT.
As long as the job at each phase in the execution meets
Maximum UUTC
SLA constraints, the total response time is enabled to
{∀user job}
satisfy the global SLA constraints. Moreover, the
introduction of
segment SLA can improve the QoS of the cloud computing subject to JQT ≤ SLAJQT
(9)
platform effectively. For instance, when a JQT violates
SLA because of a resource shortage, I/O deadlock, or JET ≤
conflicts, a higher priority is given to the job execution in
the upcoming phases, guaranteeing the resources for the job SLAJET JTT ≤
and reducing SLAJTT.
the corresponding time of JET and JTT, respectively, thus Physically, UUTC is the ratio of the operation cost and
to ensure the overall SLA of the user operation to satisfy the actual execution time. It optimizes the constraints in
the QoS constraints. An example of medical image analysis
applica- tion in cloud computing environment is shown in
Figure 2.

3.4. Utility Unit Time Cost. Currently the commercial cloud

computing platforms are mostly paid by being rented per
hour. Take the well-known Amazon EC2 cloud computing
platform for example, in which the price for standard on-
demand instances is illustrated in Table 1.
As for any user job, the Utility Unit Time Cost (UUTC)
can be defined as follows:

Total cost
UUTC = 𝑇tot . (8)
 As shown in (9), RAM makes a decision through
attaining the performance index at every observant
moment on the basis of PAM. Therefore, it is a sequential
decision-making problem. Aiming to solve this problem,
we propose a scheme in this study by employing
reinforcement learning, which is described in detail in
Section 4.

4. Resources Provisioning Mechanism

As stated in Section 3, the resource provisioning issue can
be viewed as a sequential decision-making problem in the
system model; therefore, it can be represented by MDP.
Meanwhile, with little difference in the definition for the
concepts like state space, action set, and reward function
in the various RL-based schemes of resource provisioning
in the cloud computing environment, we also define the
relevant concepts as follows in this study.
4.1.Concepts Declared
(1) Initialize 𝑄 value table
4.1.1. State Space. A physical machine can virtualize a
(2) Initialize state 𝑠𝑡
number of VMs, while a VM can only belong to one (3) error = 0
physical machine. The virtual ones are logically (4) repeat
independent within the same physical machine, while they (5) for each state 𝑠 do
compete with each other in resource provisioning. (6) 𝑎𝑡 = get action(𝑠𝑡 ) using 𝜀-greedy policy
Resources including VCPU, memory, and bandwidth, in (7) for (step = 1; step < LIMIT; step++) do
each VM, are regarded as the state space in this study; (8) take action 𝑎𝑡 observe r and 𝑆𝑡+1
(9) 𝑄𝑡 = 𝑄𝑡 + 𝛼 ∗ (𝑟 + 𝛾 ∗ 𝑄𝑡+1 −
accordingly, the state space for each VM is expressed in the 𝑄𝑡 )
form of a vector as VCPU, memory, and bandwidth. The (10) error = MAX(error | 𝑄𝑡 − 𝑄previous-𝑡 )
value of every element is not beyond the upper bound of (11) 𝑠𝑡 = 𝑠𝑡+1 , 𝑎𝑡+1 = get action(𝑠𝑡 ), 𝑎𝑡 = 𝑎𝑡+1
physical machine. Supposing a physical machine has four (12) end for
(13) end for
CPUs,8 G memory, and 100 M bandwidth, an example of
(14) Until error < 𝜃
state space in VM is (1, 2, 2), which means the VM has 1
VCPU, 2 G memory, and 2 M bandwidth.
ALGORithm 1: 𝑄 value learning algorithm.
4.1.2.Action Space. As for the 𝑖th VM resource, a possible
action space includes the increase, constancy, or decrease
in resource, which can be identified by 1, 0, −1, based on the different demands for specific resources of
respectively, to indicate the corresponding action. different jobs, when the job arrival rate increases, the uti-
Meanwhile, the increased or decreased resource of VCPU, lization strategy can make a real-time provision with VCPU
memory, and bandwidth is set asa VCPU, 512 M memory, resources in virtual machines efficiently and meanwhile
and 0.5 M bandwidth at each decision-making moment. avoid the SLA conflict as shown in Figure 3(b). However,
Then for the 𝑖th VM, assuming that its state space is (1,2 under the same experimental conditions, basic 𝑄 learning
), the action at the decision moment can be expressed as scheme adjusts VCPU resources provisioning frequently,
(0, 1, −1), which means the number of VCPU remains which may result from a wrong performance (e.g., when it
unchanged, memory increases to 512 M, and the bandwidth is the time to increase VCPU resources, they are not raised
reduces to 0.5 M. After the action is implemented, VM’s but reduced instead) due to the exploration-exploitation
state space is represented as a vector (1, 2.5, 1.5).
mechanism, eventually leading to the frequent resource
provisioning and SLA conflicts.
4.1.3. Immediate Reward. The immediate reward is used to
reflect the correct running state and the efficiency of job From the comparison results in Figure 3, we can draw
scheduling. The three situations are considered by some conclusions of basic reinforcement learning used in
designing a reward function. (1) If the UUTC of current cloud computing resources provisioning.
users job is bigger than the mean UUTC and satisfies
(i) Slow convergence.
SAL or QoS constraint, the reward is 1; (2) if the response
time of users job violates SLA or QoS constraint, the (ii) Ineffective adaptability to the changeable arrival
reward is −1; (3) otherwise, the reward is 0. rates; thus the policy needs updating, and
sometimes not getting the converged solution.
4.2. Basic Reinforcement Learning Resources Allocation (iii) The suboptimal solution is often acquired instead of
Scheme. As is depicted in the MDP, we employ the 𝑄 the optimal solution, even at a fixed job arrival rate.
learning algorithm, a popular reinforcement learning
algorithm, to solve the sequential decision problem The above disadvantages of reinforcement learning,
described in (9). The pseudocode of the basic 𝑄 value espe- cially the slow convergence rate and ineffective
learning algorithm is illus- trated in Algorithm 1. adaptive ability, severely limit its practical application in
In order to evaluate the performance of the proposed the cloud resource provisioning.
resources provisioning scheme, we compare it to the
utiliza- tion ratio provisioning scheme [16] used in the 4.3. Improved Reinforcement Learning
Amazon cloud computing platform. Due to the numerous Resources Allocation Scheme. We design an improved
experimental results, the ones listed here are only the scheme to conquer each weakness of basic
results relevant to the VCPU resource provisioning, as reinforcement learning scheme, which is focused on the
shown Figure 3. In order to make our simulations more following aspects.
convincing, we use the practical parameters and pricing
rates of Windows Azure. Windows 4.3.1. Offline Learning. With the simulations to the real data
Azure is a cloud platform developed by Microsoft, which sets, the offline training based on the basic 𝑄 learning
provides on-demand computation and bandwidth resources algorithm is employed to acquire the varied job arrival
for services through Microsoft data centers. rates, the numbers of the VMs, and the 𝑄 value table, the
We assumed the increasing arrival rate of different jobs approximate function relation between resource provision-
and the number of user jobs to be completed at each arrival ing. During offline learning process, multiple instances can
rate to be the same in the experiments. As seen in Figure run parallelly in order to learn by dividing the state space,
3(a), so
 7 1
0.9
6
0.8

SLA conflict detection (%)

0.7
5
VCPU number

0.6
4 0.5
0.4
3
0.3
0.2
2
0.1
1 0
20 40 60 80 100 120 140 160 180 20 40 60 80 100 120 140 160 180
Job number Job number
Divide State Space
Repeat Utilization scheme
for each stateBasic
space partition scheme
do Utilization scheme
Q allocation
(4 set upper and low bound of CPU, memory Basic Q allocation scheme
) bandwidth
and (a)
(b)
Obtain running state of cloud computing platform
(5e 3: Comparison
FIGUR results of index
Obtain performance VCPUforresource do and SLA conflict detection between the basic 𝑄 learning scheme and
provisioning
each resources
the )utilization scheme. (a) Comparison of 1VCPU number between the basic 𝑄 learning scheme and the utilization scheme under
using Algorithm
various
(6 job numbers. (b) SLA conflict end fordetection between the basic 𝑄 learning scheme and the utilization scheme under various job
numbers.
)
(7 for
end
)
Update 𝑄 value table
(8
) (1) Obtain running state of cloud computing platform
(9ALGORithm 2: Offline reinforcement learning algorithm.
(2) Look up 𝑄 value table, configure VM resources
) (3) Obtain performance index
(4) use belief library
(5) set upper and low bound of VCPU, memory and
that the acquired relation can be approximately formulated bandwidth
with the regression function. The pseudocode of the 𝑄 (6) action space compact
value offline learning algorithm is illustrated in Algorithm (7) for each resources do
2. (8) using Algorithm 1
In spite of the fact that the 𝑄 value table resulting from (9) end for
offline learning is rather huge, data index can be used to (10) Update 𝑄 value table
accelerate the search speed, thus improving search
efficiency.
ALGORithm 3: Online reinforcement learning algorithm.
4.3.2. Belief Libraries and Simple Action Space. Set up a
VCPU belief library, whose rules are similar to the ways
by which the belief library was built in [17]. Based on the
established belief library, the action space can be simplified belief library and the simplification of action space avoid
correspondingly. For example, when provisioning the blindness of 𝑄 learning action selection effectively and
resources, if the VCPU utilization approaches the lower thus improve the convergence speed.
bound, the action increase to raise up VCPU resources in
action space should be removed; otherwise the decrease to 4.3.3. Online Learning. Offline learning environment can
reduce the VCPU resources should be removed. The only simulate part of the real operating environment. The
establishment of acquired function is a viable strategy for resource
provision- ing only when a job arrival rate meets the SLA
constraint; even so, it may not be a suboptimal strategy.
However, this initial strategy sets up an upper boundary for
resource provisioning, and under the guidance of it we can
learn online based on this initial strategy to further improve
the resource utilization.
The acquired real-time resource utilization rate by using
PMA guides RMA learning. The pseudocode of the 𝑄-
value online learning algorithm is illustrated in Algorithm
3.
 9
1
8 0.9

7 0.8

SLA conflict detection (%)

0.7
6
VCPU number

0.6
5
0.5
4 0.4

3 0.3
0.2
2
0.1
1
20 40 60 80 0
100 120 140 160 180 20 40 60 80 100 120 140 160 180
Job number Job number

Utilization scheme Utilization scheme

Basic Q allocation scheme Basic Q allocation scheme
Improved Q allocation scheme Improved Q allocation scheme
(a) (b)

FIGURe 4: Comparison results of VCPU resource provisioning and SLA conflict detection between the improved 𝑄 learning scheme, the
basic 𝑄 learning scheme, and the utilization scheme. (a) Comparison of VCPU number between the improved 𝑄 learning scheme, the
basic
𝑄 learning scheme, and the utilization scheme under various job numbers. (b) SLA conflict detection between the improved 𝑄 learning
scheme, the basic 𝑄 learning scheme, and the utilization scheme under various job numbers.

5. Experience Results
𝑄 learning algorithm, (2) the utilization resources
To evaluate the efficiency of our approach, provision- ing scheme, denoted by utilization scheme, in
implementations have been performed on the simulation which the cloud computing resources were optimally
and real cloud computing environment, respectively. scheduled to the VMs by resources utilization, (3) the
genetic algorithm resources pro- visioning scheme, denoted
5.1.Simulation Experiment Results. Using MATLAB by GA scheme, in which the cloud computing resources
R2012a by MathWorks, Inc., we have developed a discrete were optimally scheduled to the VMs by genetic algorithm,
event simulator of the cloud server form to validate the (4) the nonlinear programming resources provisioning
efficiency of resource provisioning solution and have scheme, denoted by nonlinear programming scheme, in
compared the performance information among the which the cloud computing resources were opti- mally
alternative schemes in our simulations. scheduled to the VMs by nonlinear programming.
We evaluated the performance of the improved 𝑄 learn- As seen from the experimental results in Figure 5(a),
ing strategy in Figure 4 and compared it with the utiliza-
the number of VCPU resources in the GA scheme and the
tion rate scheme and the basic 𝑄 learning scheme under
the same experimental conditions in Section 4.2. From the nonlinear programming scheme may cause frequent
experimental results in Figure 4, on one hand, we can see changes due to the objective function optimization.
that basic 𝑄 learning scheme is still likely to make wrong Nevertheless, the improved 𝑄 learning scheme and the
decisions and thus resulted in the SLA conflicts and utilization scheme demonstrate the same performances as
frequent VCPU resource provisions. On the other hand, the those in Figure 4(a), respectively. With similar pricing
improved settings to Table 1, Figure 5(b) shows that the total cost of
𝑄 learning scheme can provision the VCPU resources in the improved learning
real time based on the changes in arrival rates; more 𝑄 strategy designed in this paper is lower than that of the
importantly, apart from avoiding SLA conflicts, the number compared schemes.
of VCPU resources used by it appears less than that used by We ran the simulation program 1000 times at various
utilization rate scheme through most of the time. In other arrival rates, and the average results of various performance
words, avoiding the SLA conflict, the improved 𝑄 learning indicators are shown in Table 2. As shown in the results, the
method improves the utilization rate of resources. UUTC at different arrival rates is better than that of the
Next, we compare the improved 𝑄 learning scheme con- trast algorithm. In light of the definition of UUTC, the
with the prevalent resource provisioning schemes: (1) the numerator aims to maximize the current operation costs,
proposed resources provisioning scheme, denoted by the while the denominator aims to minimize the current execu-
improved 𝑄 learning scheme, in which the cloud tion time. The ratio between the two indicates the
computing resources were optimally scheduled to the VMs utilization rate of cloud computing resources per unit time.
by improved Accordingly, the greater the value is, the higher the
corresponding utiliza- tion rate of the cloud computing
resource is.
 TABLE 2: Comparison results of various resources allocation
schemes.

Arrival rate Total running time Total cost Mean VCPU numbers UUTC
(jobs per min.) (min.) (dollars) (numbers per arrival rate) (dollars per min.)
RPSUT 364.61 7270 7.36 2.70
RPSGA 381.68 3930 7.36 1.39
30
RPSNP 368.36 4353 7.36 1.60
Proposed scheme 368.36 5204 4.76 2.96
RPSUT 247.20 7433 8.20 3.66
RPSGA 273.12 5788 8.20 2.58
40
RPSNP 278.21 5349 8.20 2.34
Proposed scheme 250.30 3727 3.87 3.84
RPSUT 216.14 8688 8.18 4.91
RPSGA 244.06 8100 8.18 4.05
50
RPSNP 232.41 6376 8.18 3.35
Proposed scheme 218.63 5448 4.45 5.59
RPSUT 148.39 9540 8.78 7.32
RPSGA 172.75 8718 8.78 5.74
60
RPSNP 176.30 7569 8.78 4.88
Proposed scheme 149.85 6578 4.43 9.90
RPSUT 147.48 9611 9.96 6.54
RPSGA 175.78 10025 9.96 5.72
70
RPSNP 179.37 9878 9.96 5.529
Proposed scheme 147.87 7820 7.69 6.87

20
11000
18
10000
16

14 9000
Total cost (dollars)
VCPU number

12 8000
10
7000
8

6 6000

4 5000
2
4000
20 40 60 80 100 30 35 40 45 50 55 60 65 70
Job number Arrival rate (jobs/min.)

Utilization scheme Utilization scheme

Improved Q learning scheme Improved Q learning scheme
GA scheme GA scheme
Nonlinear programming scheme Nonlinear programming scheme
(a) (b)

FIgURE 5: Comparison results of VCPU resource provisioning and total cost between the improved 𝑄 learning scheme, GA
provisioning scheme, the utilization scheme, and nonlinear programming provisioning scheme. (a) Comparison of VCPU number
between the improved
𝑄 learning scheme, GA provisioning scheme, the utilization scheme, and nonlinear programming provisioning scheme under various job
numbers. (b) Comparison of total cost between the improved 𝑄 learning scheme, GA provisioning scheme, the utilization scheme, and
nonlinear programming provisioning scheme under various job arrival rates.
 TABLE 3: Comparison results of average throughput under various
12000 warehouses (Bops).
0 Utilization
Warehouse Max Proposed
10000
scheme scheme
Scores

0
1 32103.43 32847.88 33767.96
80000
2 63252.47 59510.64 59951.13
60000
3 87267.56 82380.99 83919.59
40000
4 105719.64 104081.37 99212.12
20000 4 8 5 115648.76 117200.12 116671.41
Warehouses 6 119057.39 121530.59 122071.97
Included in score calculation 7 123759.19 126187.14 124289.81
Not included in score calculation 8 120844.70 122457.18 122758.30
FIGURe 6: SPECjbb2005 throughput in default settings.
8

The utilization scheme in the figure executes the jobs 7

with the real-time resource provisioning based on the
utilization. It is demand-sensitive to the resources and
6
requires frequent application and release of the resources.
Based on the job arrival rates, the improved 𝑄 provisioning CPU number
scheme can not only configure the system resources in 5
accordance with 𝑄 value table optimization scheme, but
also learn the optimal provisioning further on the real-time 4
resource utilization. The improved 𝑄 allocation scheme can
configure the cloud computing resources based on the
3
arrival rates of the user’s job and learn the optimal
allocation further based on the opti- mization scheme. This
scheme facilitates the adaptability to the user’s job arrival 2
100 120 140 160 180
20 40 60 80
rates, while the basic 𝑄 learning scheme
needs some exploration before it gets steady, or even worse, it Time (min)
cannot acquire the stable distribution. From the experiment
Utilization scheme
results in Figure 5, a similar conclusion is drawn, which Improved Q learning scheme
further proves the effectiveness of this provisioning scheme.
FIgURE 7: Comparison of CPU number between the improved
𝑄 learning scheme and the utilization scheme under various
5.2.Real Cloud Computing Platform Experiments Results. warehouses.
The machines used in the experiments consist of virtual
servers, client, and compute machines. The physical
machines for virtual hosting are Lenovo ThinkServer
RD630 with 8 CPU and 16 GB memory. dynamically increasing from 1 to 8 every fixed time
Xen was used as our virtualization environment and interval. In our experiments, the time interval was set as 20
the SPECjbb2005 was selected as the workloads running minutes. Another fact discovered in the experiments at the
within the VMs. SPECjbb2005, a Java program, is SPEC’s same time was that the benchmark was insensitive to
benchmark. By simulating a three-tier client/server system memory. Hence, only the results of CPU resources were
with stress on the middle tier, the benchmark measures the listed in the next part. We ran the benchmark 10 times at
performance of server side Java. It implements the Java various warehouses, and the comparison results of average
virtual machine (JVM), just-in-time (JIT) compiler, garbage throughput under various warehouses were shown in Table
collec- tion, threads, and part of the operating system. 3. From the results we can see the throughput of the
Implemented in a more object-oriented manner, utilization scheme and the proposed scheme was close to
SPECjbb2005 presents new features such as XML the reference Max under various warehouses. It proves
processing and BigDecimal com- putations and furnishes a these two resource provisioning schemes can provision
new enhanced workload to mirror how real application is CPU resources self-adaptively and achieve the maximum
designed, thus making the benchmark a more realistic throughput according to different
reflection of contemporary application [18]. warehouses.
The SPECjbb2005 throughput under various pressures The real cloud computing environment experiment
in the default setting can be seen in Figure 6. The default results as shown in Figure 7 also demonstrated that the
setting in SPECjbb2005 refers to the whole resources in the proposed improved 𝑄 provisioning scheme outperforms
platform used by the system under any pressure, such as the utilization scheme in that it achieved resources
CPU and memory. In the experiments processing, utilization rate under various warehouses and CPU
warehouses were resources constraint.
6. Conclusions and Future Work SLA: Service Level Agreement

In this study, we provide an insightful view about the

resource provisioning optimization problem in cloud
computing plat- form; then we propose a novel resources
provisioning scheme based on the reinforcement learning
and queueing theory. With the introduction of the concepts
of SSLA and UUTC, we view the resource provisioning
issue in cloud computing as a sequential decision problem,
and then we design a novel opti- mization object function
and employ reinforcement learning to solve it. Experiment
results not only demonstrate the effec- tiveness of the
proposed scheme, but also prove to outper- form the
prevalent resource provisioning methods in terms of SLA
collision avoidance and user costs. Also, some conclu-
sions can be drawn as below for using 𝑄 learning
algorithm.

(i) The 𝑄 learning algorithm outperforms the compara-

tive method in three aspects: the number of CPUs in
use, SLA conflicts, and costs.

(ii) Belief library is employed to simplify the action

space, thus to enhance the performance of the 𝑄
learning algorithm.

(iii) State space is the number of VCPU/CPU in experi-

ment, with a smaller space, faster convergence, and
stronger adaptability.

(iv) 𝑄 learning provisions in two cases, either when the

SLA violation occurs for three consecutive times, or
when the utilization rate is too high or too low.

In the future, we plan to extend our schemes for the

dynamic resource provisioning to get the minimal response
time, thus providing highly satisfactory services and
avoiding SLA violations. Looking into the cloud entities
and consid- ering the details of cloud computing platform,
such as VMs failures, VMs migration, costs of
communication, and burst arrivals of requests, VMs cluster
for different kinds of requests will be other dimensions of
extension.

Abbreviations
UI: Users interface
UJQ: Users job queue
UJS: Users Job
Scheduling UJT:
Users Job Transmit
VM: Virtual Machine
VMC: Virtual Machine Cluster
VMCA: Virtual Machine Cluster
Agent PMA: Performance Monitor
Agent RMA: Resource Management
Agent JRT: Job Response Time
JQT: Job Queueing Time
JET: Job Execution
Time JTT: Job Transfer
Time QoS: Quality of
Service
SSLA: Segmentation SLA
UUTC: Utility Unit Time Cost
MDP: Markov decision process
PDF: Probability Density
Function ML: Machine learning
RL: Reinforcement learning
𝜆: Job arrival rate of cloud computing platform
𝜆 𝑖𝑗 : Job arrival rate of 𝑗th VM in 𝑖th VMC
𝜇𝑖𝑗 : Job server rate of 𝑗th VM in 𝑖th VMC
𝐷𝑖𝑗 : The executed result size of user job
𝐵𝑖𝑗 : The allocation bandwidth resources of
the user job.

Conflict of Interests
The authors declare that there is no conflict of interests
regarding the publication of this paper.

Acknowledgment
The work presented in this paper was supported by
National Natural Science Foundation of China (nos.
61272382 and 61402183).

References
[1] H.-S. Wu, C.-J. Wang, and J.-Y. Xie, “TeraScaler ELB-an
algorithm of prediction-based elastic load balancing resource
management in cloud computing,” in Proceedings of the 27th
International Conference on Advanced Information Networking
and Applications Workshops (WAINA ’13), pp. 649–654, IEEE,
Barcelona, Spain, March 2013.
[2] Y. Gao, H. Guan, Z. W. Qi, T. Song, F. Huan, and L. Liu,
“Service level agreement based energy-efficient resource
management in cloud data centers,” Computers and
Electrical Engineering, vol. 40, pp. 1621–1633, 2013.
[3] V. Suresh, P. Ezhilchelvan, and P. Watson, “Scalable and
respon- sive event processing in the cloud,” Philosophical
Transactions of the Royal Society A, vol. 371, no. 1983, Article
ID 20120095, 2013.
[4] X. Nan, Y. He, and L. Guan, “Queueing model based
resource optimization for multimedia cloud,” Journal of
Visual Commu- nication and Image Representation, vol. 25,
no. 5, pp. 928–942, 2014.
[5] H. Nguyen, T. N. Minh, and N. Thoai, “Tool-driven
strategies for resource provisioning of single-tier web
applications in clouds,” in Proceedings of the 5th
International Conference on Ubiquitous and Future
Networks (ICUFN ’13), pp. 795–799, July 2013.
[6] S. Yu, Y. Tian, S. Guo, and D. O. Wu, “Can we beat DDoS
attacks in clouds?” IEEE Transactions on Parallel and
Distributed Systems, vol. 25, no. 9, pp. 2245–2254, 2014.
[7] K. Salah and R. Boutaba, “Estimating service response time
for elastic cloud applications,” in Proceedings of the 1st
IEEE International Conference on Cloud Networking
(Cloud Net ’12), pp. 12–16, Jussieu, Paris, November 2012.
[8] H. Khazaei, J. Misic, and V. B. Misic, “Performance
analysis of cloud computing centers using M/G/m/m+r
queuing systems,” IEEE Transactions on Parallel and
Distributed Systems, vol. 23, no. 5, pp. 936–943, 2012.
 [9] H. Wada, J. Suzuki, Y. Yamano, and K. Oba, “Evolutionary
deployment optimization for service-oriented clouds,” Soft-
ware: Practice and Experience, vol. 41, no. 5, pp. 469–493,
2011.
[10] M. Bourguiba, K. Haddadou, I. E. Korbi, and G. Pujolle,
“Improving network I/O virtualization for cloud computing,”
IEEE Transactions on Parallel and Distributed Systems, vol.
25, no. 3, pp. 673–681, 2014.
[11] Z. Luo and Z. Qian, “Burstiness-aware server consolidation
via queuing theory approach in a computing cloud,” in
Proceedings of the 27th IEEE International Symposium on
Parallel & Dis- tributed Processing (IPDPS ’13), pp. 332–341,
IEEE, Cambridge, Mass, USA, May 2013.
[12] F.-C. Jiang, C.-T. Yang, C.-H. Hsu, and Y.-J. Chiang,
“Optimiza- tion technique on logistic economy for cloud
computing using finite-source queuing systems,” in
Proceedings of the 4th IEEE International Conference on
Cloud Computing Technology and Science (CloudCom ’12),
pp. 827–832, December 2012.
[13] G. Grimmett and D. Stirzaker, Probability and Random Pro-
cesses, Oxford University Press, 3rd edition, 2010.
[14] Y. Wu, C. Wu, B. Li, X. Qiu, and F. C. M. Lau, “CloudMedia:
when cloud on demand meets video on demand,” in
Proceedings of the 31st International Conference on
Distributed Computing Systems (ICDCS ’11), pp. 268–277,
IEEE, Minneapolis, Minn, USA, July 2011.
[15] J. Zheng and E. Regentova, “Qos-based dynamic channel
allocation for GSM/GPRS networks,” in Network and Parallel
Computing, vol. 3779 of Lecture Notes in Computer Science,
pp. 285–294, Springer, Berlin, Germany, 2005.
[16] A. Wolke and G. Meixner, “Twospot: a cloud platform for
scaling out web applications dynamically,” in Proceeding of
3rd European Conference on Service Wave, pp. 13–24, Ghent,
Belgium, 2010.
[17] X. Bu, J. Rao, and C.-Z. Xu, “A reinforcement learning
approach to online web systems auto-configuration,” in
Proceedings of the 29th IEEE International Conference on
Distributed Computing Systems Workshops (ICDCS ’09), pp.
2–11, Montreal, Canada, June 2009.
[18] http://www.spec.org/jbb2005/.