This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 1

A Cost-optimized Resource Provisioning

Policy for Heterogeneous Cloud Environments
Xin Chen, Feng Ding, Tiantian Zhang, Gang Hou, and Lan Lan,

Abstract—To avoid the drawbacks of a pricing mechanism in heterogeneous cloud environments that considers only single
resources, we propose a multi-resource combinatorial pricing mechanism in this paper. This approach jointly considers the
principal resources (i.e., CPU, memory, storage and bandwidth) with the goal of minimizing the total cost. Then, a cost-optimized
resource provisioning policy (CORP P ) based on game theory is applied to this mechanism that considers the Nash equilibrium
between cloud users as well as the Stackelberg equilibrium between users and the cloud provider. The experimental results
show that the proposed combinatorial pricing mechanism is more suitable in a heterogeneous cloud environment when running
various types of cloudlets. Moreover, CORPP is effective in reducing users’ total costs when using both random cloudlets and
PlanetLab cloudlets.

Index Terms—heterogeneous cloud; combinatorial pricing; resource provisioning; game theory; Nash equilibrium; Stackelberg
equilibrium.

1 I NTRODUCTION resource insufficiency during peak demand periods for vir-

computing is a recent business model. With the tual machines in a federated cloud environment. Moreover,
C LOUD
development of smart computing environments, cloud
computing can be considered as a natural improvement on
most articles based on the market economy model focus
on maximizing the benefits accrued by the CSP without
parallel computing, distributed computing and grid comput- considering the purchasing costs of CUs.
ing. The resources at cloud service providers (CSP) owns However, as mentioned before, the tasks executing on
are leased by different cloud users (CU), who can apply and a computing platform rely on different resources. Conse-
release their cloud computing resources dynamically [1-3]. quently, a single-resource pricing mechanism has natural
Because there are different types of cloudlets, the resources drawbacks: it can lead to higher costs for some CUs or
a task relies on are also vary. For example, a calculation cause unnecessary purchases. There are two possible rea-
task relies on a CPU more than on other resources, a sons for this phenomenon: (1) multiple CUs can share cloud
matrix operation may focus primarily on memory, a cloud computing resources, but traditional resource provisioning
storage service depends on available storage and bandwidth policies do not consider the impact on CUs when using
resources, and so on. common resources; (2) existing studies based on market
Resource pricing is a foundational concern when s- economy game theory do not consider the influence of CSP
tudying a cloud computing provisioning policy. However, pricing strategy or resource provisioning policy change on
researchers have mostly considered only single-resource CUs [7,8].
pricing mechanisms (i.e., for CPU, memory or network). Therefore, the sustainability of cloud computing sys-
For example, Li et al. [4] proposed a scheduling strategy to tem urgently requires a multi-dimensional resource pricing
maximize the benefits of a CSP’s network resources based mechanism for heterogeneous cloud computing resources
on a network resource bidirectional auction mechanism in and a resource provisioning policy that simultaneously
a federated selfish cloud environment. Many similarities considers different CUs and the interplay between CUs
exist between the cloud computing environment and the and the CSP. In this paper, we propose a multi-dimensional
free market in economic theory; thus, many scholars have resource combinatorial pricing mechanism that jointly con-
proposed methods based on market mechanisms to solve siders the principal resources (i.e., CPU, memory, storage
the cloud computing provisioning problem. Goiri et al. [5] and bandwidth) with the goal of minimizing the total cost.
proposed a federated cloud architecture for CSPs based Furthermore, a cost-optimized resource provisioning pol-
on economics. Toosi et al. [6] proposed an approach to icy (CORPP) in a heterogeneous cloud environment is
outsource or terminate cloud tasks to solve the problem of defined based on game theory that maximizes system utility
while minimizing the total cost to CUs.
• X. Chen is with the School of Electronics and Information Engineering, The rest of this paper is organized as follows. In Sec-
Liaoning University of Technology, Jinzhou, China. tion 2, we introduce the resource pricing mechanism and
• F. Ding, T. Zhang and G. Hou are with the School of Software
Technology, Dalian University of Technology, Dalian, China; email:
provisioning policy and then analyze the game relations
[email protected] in CORPP in Section 3. The experiments and results are
• L. Lan is with the International Studies of College, Shenyang University, presented in Section 4, and final conclusions and future
Shenyang, China.
research directions are given in Section 5.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 2

2 R ESOURCE PRICING MECHANISM AND combined price of CPU, memory, storage, and bandwidth
PROVISIONING POLICY for server Hi .
2.1 Problem description
2.3 Cost-optimized resource provisioning policy
In a cloud computing system, CUs submit resource require-
(CORP P )
ments and different types of cloud tasks to a CSP. The cloud
computing resource distributor consists of a pricing mecha- The global resource provisioning policy of a CSP is defined
nism, a statistical mechanism, a task scheduler, and a virtual as Φ = (ϕ1 , ϕ2 , ..., ϕn )T and determines the provisioning
machine manager. The pricing mechanism sets the price for of CU’s resources. In CORP P , ϕj is the policy of Uj
multi-dimensional combinations and is responsible for pric- (1 ≤ j ≤ n), expressed as (rj1 , rj2 , ..., rjm )T , in which
ing different types of resources from different cloud servers. rji = (cpuij , memij , stgeij , bwji ) (1 ≤ i ≤ m). The total
The pricing mechanism is responsible for recording each resources required by all CUs on server H∑ i is denoted by
n
CU’s purchases and the revenue generated by each cloud Ri = (T cpui , T memi , T stgei , T bwi ) = j=1 rji , and is
server. The task scheduler is responsible for scheduling restricted to be no greater than the maximum capacity of
different cloud tasks. CORPP includes two types of cloud the corresponding resources of RESi , i.e.,
tasks: (1) User tasks (those submitted by a CU); and (2) 

 T cpui ≤ CP Ui
System tasks (those required to manage the normal opera- 
T memi ≤ M EMi
tion of cloud computing system). Different tasks running on (1)

 T stgei ≤ ST GEi
different cloud servers will affect a CU’s purchasing costs. 
T bwi ≤ BWi
The virtual machine manager is responsible for dynamically
creating, starting, suspending, and deleting virtual machines where 1 ≤ i ≤ m. The function set of the server’s cost is
based on the CUs’ resource requirements. A cloud data defined as
center is heterogeneous, containing a variety of different
types of cloud servers that provide CPU, memory, storage, F = {F1 (·), F2 (·), ..., Fm (·)}, (2)
network bandwidth and other physical resources required
to run the virtual machines. The structure of a cloud in which Fi (rji ) represents the cost of Uj using CPU,
computing system is shown in Figure 1. memory, storage, and bandwidth resources rji on server
The cloud task scenario we consider in this paper is as Hi , and the cost of all CUs generated on the server Hi is
follows. Multiple CUs submit resource requests as well denoted by Fi (Ri ). From the above definition we can see
as cloud tasks to the CSP at the same time. During the that for policy Φ to be feasible, we must meet the following
duration of a lease period, all the cloud tasks from CUs conditions:
can be completed. Thus, we can make the following three  ∑m i
assumptions: (1) all the resource types in the system (e.g.,  i=1 rj = ωj
rji ≥ 0 (3)
CPU, memory, storage, and bandwidth) are independent; 
(2) The CSP has a sufficient number of resources to meet RESi − Ri ≥ 0
the resource requirements of all its CUs within a lease for each i ∈ [1, m] and j ∈ [1, n].
period; and (3) each CU behaves selfishly; the CSP wishes Formula 3 represents the fact that the sum of all server
to retain its CUs by offering relatively low prices to improve resources assigned to Uj should be equal to Uj ’s demand,
its reputation in the cloud computing market and, thus, i.e., ωj , and that the total demand on each resource should
attract more CUs. be limited to the maximum capacity of the corresponding
resource of each server.
2.2 Multi-dimensional resources combinatorial Then, the price that Uj pays is the total cost of all the
pricing mechanism server resource Uj uses, i.e.,

Assume that a heterogeneous cloud data center has m ∑m

U CΦ (j) = ψ(i, j) · Fi (Ri ). (4)
cloud servers, denoted as H = {H1 , H2 , ..., Hm }. The i=1
maximum CPU, memory, storage and bandwidth capac- Here, ψ(i, j) = 1 when Uj uses server Hi ; conversely,
ity of each server Hi is defined as a vector RESi = ψ(i, j) = 0. Because Formula 4 presents the purchasing
(CP Ui , M EMi , ST GEi , BWi ). U = {U1 , U2 , ..., Un } cost of Uj , maximizing −U CΦ (j) means that Uj maxi-
denotes n CUs all using the cloud computing resources mizes its utility function.
simultaneously. The resource requirement of U is Ω = The cost of the entire cloud computing system is deter-
(ω1 , ω2 , ..., ωn )T , in which the CPU, memory, storage, and mined by the total costs T CΦ of the global provisioning
bandwidth requirements of Uj are expressed as ωj = policy Φ, i.e.,
(cpuj , memj , stgej , bwj ), respectively. For different types
∑m
of cloud servers, the prices of different resource types vary; T CΦ = Fi (Ri ). (5)
the multi-dimensional resource combinatorial pricing in this i=1

paper is accurate for the four resources of each cloud server. Using the same logic, maximizing −T CΦ means that we
We define P (RESi ) = (Picpu , Pimem , Pistge , Pibw ) as the obtain the smallest total system cost.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 3

Fig. 1. The structure of a cloud computing system

3 G AME THEORY IN CORP P can be achieved in CORP P , we define the cost-optimized

problem as follows:
Market economic game theory can be used to analyze
the relationships between multiple CUs as well as the
∑n ∑m T
relationship between CUs and the CSP with the goal of M inimize T CΦ = rji · (P (Ri )) (6)
finding the most suitable resource provisioning policy for j=1 i=1

the cloud computing system. The relationships in CORP P s.t.,  ∑m i

are shown in Figure 2, where the CSPs are responsible  i=1 rj = ωj
for the cloud server and for selecting the CPU, memory, rji ≥ 0 (7)

storage and bandwidth resources required to run the cloud RESi − Ri ≥ 0
computing system, and the CUs apply to use the required where 1 ≤ i ≤ m and 1 ≤ j ≤ n.
resources. Because the resources are shared by CUs, there Furthermore, Formula 6 can be re-written as follows:
is a competition between multiple CUs. Moreover, there is ∑m T
also a cost game between the CUs and the CSP. T CΦ = Ri · (P (Ri )) (8)
i=1

According to Formula 2, we have

3.1 Game between multiple CUs Fi (Ri ) = Ri · (P (Ri ))T
∑n T
In CORP P , a user Uj adjusts its strategy selfishly; max- = j=1 rji · (P (Ri ))
imizing the utility function −U CΦ (j) minimizes its own ∑n
= j=1 (cpuij · Picpu + memij · Pimem + stgeij · Pistge + b
costs. However, because multiple CUs share cloud com- (9)
puting resources, this strategy increases the costs U CΦ (k) Which leads to the following theorem.
of other Uk (k ̸= j). Because CUs are selfish, each CU ∑n T
Theorem 1: Fi (Ri ) = j=1 rji · (P (Ri )) is a convex
tries to minimize its own purchasing costs; thus, the CUs function on Rn .
continually change their strategies until they can no longer Proof :
reduce the cost. This dynamic adjustment continues until Let x, y ∈ Rn and α ∈ [0, 1], then we have
the system reaches a stable state, i.e., no CU can reduce
costs by changing its strategy unilaterally. This stable state
is known as a Nash equilibrium. F
∑i [α · rxi + (1 − α) · ryi ]
n
= [α · rxi + (1 − α) · ryi ] · (P (Ri ))T
Definition 1: A feasible global resource provisioning ∑x,y=1
n T ∑n T
x=1 α · rx · (P (Ri )) + y=1 (1 − α) · ry · (P (Ri ))
i i
policy Φ = (ϕ1 , ϕ2 , ..., ϕn )T can be considered as having =
achieved Nash equilibrium when the following condition is = αFi (rxi ) + (1 − α)Fi (ryi ),
met; this provisioning policy is named the Nash Provision- Then, according to the definition of a convex function,
ing policy. we have proved this theorem.
′
If Uj ∈ U , and for any other Φ , we have U CΦ (j) ≤ Therefore, the above nonlinear optimization problem is
U CΦ′ (j), and U CΦ (k) = U CΦ′ (k) (k ̸= j). a convex programming problem, and any local optimal
In the Nash equilibrium state, no CU will change its cur- solution is a global optimal solution. Therefore, Formula
rent policy. However, although Nash equilibrium represents 8 can be further rewritten as follows:
a stable state of the system, the total costs T CΦ may not ∑m
be optimal. To explore the best possible performance that T CΦ = Fk (Rk ). (10)
i=1
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 4

Fig. 2. Game relationship in CORPP

The representation method and the constraint conditions the CUs can be seen as the participants, who selfishly
of T CΦ are similar to the formula for solving the selfish attempt to reduce their own purchasing costs.
1 2 m T
routing [9]. One needs only to make the corresponding We use ΦL = (rL , rL , ..., rL ) to represent the resource
change: replacing the ”path” in selfish routing in [9] with provisioning policy of the CSP (i.e., the leader’s policy in
i
”cloud users” in CORP P , and using ”physical resources” the system), where rL represents the CPU, memory, stor-
in CORP P to represent the ”side” in [9]. Finally, to obtain age, and bandwidth resources used by the leader on server
the ”global resources provisioning policy” in CORP P , Hi . The total amount ∑m ofi resources used by the leader is
we need to find the ”flow” in [9]. Then, the equivalence defined as ωL = i=1 rL . The Stackelberg global resource
between CORP P and selfish routing can be derived from provisioning policy is defined as (ϕ1 , ϕ2 , ..., ϕn , ϕL )T , the
their respective optimization formulas. When T CΦ uses this total amount
∑n of resources used on server Hi is RiS =
equivalence, based on T heorem 3.2 in [9], we obtain the rL + j=1 rj , which must meet RESi − Ri s ≥ 0.
i i

following theorem. Therefore, each CU can see that the resources on the
Theorem 2: When the various resource cost functions of server Hi are reduced due to the leader’s use, which affects
(H, F, Φ) are continuously non-decreasing, CORP P can their equilibrium. The leader’s ωL can be assigned to any
reach the Nash equilibrium. Furthermore, the Nash provi- task that he controls, such as user task scheduling and
sioning policy of CORP P involves solving the following management, infrastructure management, and so on.
optimization problem to minimize T CΦ under Constraint A Stackelberg game achieves equilibrium in the fol-
7, in which the server cost function is lowing circumstance: the leader selects a strategy and
the followers cannot unilaterally change their strategies to
∫ cpu ∫ memi mem reduce their costs. The definition of Stackelberg equilibrium
Fi (Ri ) = Ri · ( 0 i ficpu (t)dt, 0 fi (t)dt,
∫ stgei stge ∫ bwi bw in SCORP P is as follows.
0
f i (t)dt, 0
f i (t)dt), Definition: For SCORPP (H, F, Φ, ΦL ), the Stack-
∫ cpui cpu (11) elberg global resource provisioning policy ΦS =
where 0 fi (t)dt represents the functional relation (ϕ1 , ϕ2 , ..., ϕn , ϕL )T , if Φ = (ϕ1 , ϕ2 , ..., ϕn )T is the
between the price and the time of the CPU on server Hi . Nash equilibrium for CORP P (H, F N , Φ), where F N =
The other three integral functions are similar. {FiN |FiN (x) = Fi (x + rL i
)}, then ΦS achieves Stackelberg
equilibrium. The complete provisioning policy is called the
Nash-Stackelberg provisioning policy.
3.2 Game between CUs and CSP
When the cost function assigned by the Stackelberg
To overcome the local optimal in Nash equilibrium for policy is the lowest, then the resource provisioning policy
arbitrary cost functions, we consider a variant of the Nash is optimal. For best performance, Stackelberg equilibrium
equilibrium in CORP P , i.e., the Stackelberg equilibrium. relies on the leader’s strategy; in other words, the leader’s
In this case, the game includes two types of players: a strategy also determines the system’s performance. In this
leader, whose goal is to optimize the utility of the whole paper, we consider two kinds of leader strategies: the Aloof
system, and a group of participants, who are selfish; each strategy and the Least Cost First (LCF) strategy [10].
type attempts to maximize its own utility. The leader cannot Aloof strategy: In this strategy, the CSP ignores the
directly determine the participants’ strategies; however, the CUs’ strategies; instead, it tries to find a provisioning
leader can influence the followers’ strategies through his or strategy for ΦL in which ΦL is optimal for CORP P
her own strategy. Therefore, the leader wants to choose a (H, F, ΦL ), where ΦL = {ωL }. In the Aloof strategy,
strategy that causes the system to achieve an approximately the leader does not consider the followers’ strategies; he
optimal solution. Consequently, CORP P can be modeled determines the resource provisioning policy independently.
after Stackelberg equilibrium, where the CSP plays the LCF strategy: In the LCF strategy, the leader uses the
leader role and tries to maximize the system utility, while resources that other clients tend to reject: those whose
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 5

prices are higher. The following algorithm is used to find pricing result is sufficiently high (which we define as 60%),
the LCF strategy. we say the CPU-pricing mechanism is sufficient because
leftmargin=*,labelsep=3mm CPU pricing plays a prime role, and the single-resource
1) The leader first solves Formula 6 to obtain the opti- pricing mechanism is much simpler than the combinatorial
mal provisioning policy Φ∗ of CORP P (H, F, Φ∗ ), pricing mechanism. The experiments are run several times
recorded as Φ∗ = (ω1 , ω2 , ..., ωn , ωL )T . This allo- and the average is taken as the result.
cation is not necessarily balanced for CUs; that is, Figure 3 shows the total costs of the two pricing mecha-
some Uj may reduce their costs by changing their nisms when employed with different numbers of users and
strategies. four types of virtual machines running CPU-intensive tasks.
2) Sort the resources to make F1 (Φ∗ (1)) ≤ When using M-VM with the lower CPU configuration, the
F2 (Φ∗ (2)) ≤ ... ∑ ≤ Fm (Φ∗ (m)) and find the total cost based on CPU pricing is approximately 42% of
smallest i∗ , so that i=i∗ Φ∗ (i) ≤ ωL . Here, Φ∗ (i)
m the total cost based on combinatorial pricing. But as the
represents the resources that the optimal provisioning CPU configuration in the virtual machines improves (i.e.,
policy Φ∗ allocates on server Hi . when using S-VM, E-VM or H-VM), the proportion of
3) The leader’s strategy ΦSL ∗ = (rL 1 2
, rL m T
, ..., rL ) is the total costs based on the CPU to the total costs based
 ∗as follows:
determined on combinatorial pricing gradually increases, from 60%, to
 Φ (i),∑ i > i∗ 73%, and finally, to 78%, respectively. This result indicates
ωL − k=k∗ Φ (k), k = k ∗
∗
i m
rL = that when the cloud task is using more powerful CPU

0, k < k∗ configurations such as H-VM, it is reasonable to use the
In the LCF strategy, the leader adopts the (m − k ∗ ) CPU-pricing mechanism as the pricing method. In contrast,
servers with the highest costs, which induces the followers when using a less-powerful CPU configuration such as M-
to achieve Nash equilibrium. According to T heorem 4.2 in VM, the prices under this single-resource mechanism are
[10], the following conclusions can be obtained. When ΦSL less than half the total costs generated by the combinatorial
is the LCF strategy of CORP P (H, F, Φ)’s leader, which pricing. This result shows that CPU cannot be the only
includes one method that provides ΦSL for the followers’ condition considered by a pricing strategy.
Nash and another one providing Φ∗ for the CORP P Similar scenarios occur in Figure 4, in which network
(H, F, Φ)’s optimum, then CT(ΦSF ,ΦSL ) ≤ α1 CTΦ∗ , where I/O-intensive tasks are being executed. The ratios between
α represents the proportion that the leader’s tasks account the CPU-pricing results and the combinatorial pricing re-
for of the entire system task. This indicates that the worst sults are 35%, 47%, 51%, and 40% when using M-VM,
case for the LCF policy is α1 times the optimal provisioning S-VM, E-VM, and H-VM configurations, respectively. This
policy. result indicates that the single CPU-pricing mechanism
is completely unsuitable for pricing network I/O-intensive
4 E XPERIMENT AND RESULTS tasks.
4.1 Experiment configuration From the above analysis, the CPU-pricing mechanism
considers only one resource; consequently, it may be suit-
The proposed CORP P algorithm is implemented on the
able for CPU-intensive tasks, but cannot be applied to other
CloudSim 3.0.3 platform [11]. The experiment simulates
types of tasks, such as I/O-intensive tasks. Therefore, the
four types of virtual machines commonly used by Amazon
multi-dimensional resource combinatorial pricing mecha-
EC2, namely, H-VM, E-VM, S-VM and M-VM, whose
nism based on the CPU, memory, storage and network
configurations are shown in Table 1. The cloud data center
bandwidth and proposed in this paper is more suitable for
has 100 servers; 50 are HP ProLiant ML110 G4s, and the
pricing cloud computing resources.
others are G5s. The G4s are configured with a Xeon-3040
For our combinatorial pricing mechanism experiments,
@ 1860 MHz, 2 cores, and 4 GB of RAM, while the G5s
we executed randomly generated tasks and PlanetLab cloud
contain a Xeon-3075 @ 2660 MHz, 2 cores, and 4 GB of
tasks and then compared the total costs using four resource
RAM.
provisioning policies: the optimal provisioning policy, Nash
Our experiments use two types of cloud tasks: CPU-
provisioning policy, Nash-Stackelberg provisioning poli-
intensive tasks and PlanetLab tasks that can be considered
cy and Round-Robin policy. Moreover, we compared the
as network I/O-intensive tasks [12,13]. The experimental
highest income obtained by a single cloud server under
simulation cycle is 24 h; for all cloud tasks, the MIPS range
these four policies. Among these four resource provisioning
is between 2.0 ∗ 107 and 4.0 ∗ 107 .
policies, the first three are based on game theory, where the
optimal provisioning policy is theoretically optimal, while
4.2 Results analysis the leader in the Nash-Stackelberg provisioning policy uses
The experiment first compares the difference between a the LCF strategy, with α = 0.1. The last policy, Round-
CPU-pricing mechanism and the combinatorial pricing Robin, is not a game-based policy.
mechanism with different types of virtual machines and Figure 5 and Figure 6 respectively show the total costs
different quantities of CUs when running either CPU in- and highest income of a single server with randomly
tensive tasks or network I/O-intensive tasks. When the generated tasks generated by four different provisioning
ratio between the CPU-pricing result and the combinatorial policies under the combinatorial pricing mechanism. From
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 6

TABLE 1
Four types of virtual machines used by Amazon EC2

VM types CPU (MIPS) Memory (GB) Storage (GB) Bandwidth (MB)

H-VM 2500 0.85 100 100
E-VM 2000 3.75 100 1000
S-VM 1000 1.7 100 1000
M-VM 500 0.633 100 100

Fig. 3. The total costs of CPU intensive tasks

Fig. 4. The total costs of I/O intensive tasks

Figure 5, the highest total cost is generated by the Round- resources and the selfish characteristics of CUs. In addition,
Robin provisioning policy, while the results from the Nash the Round-Robin provisioning policy does not consider the
and Nash-Stackelberg provisioning policies are both close games between the CUs, or the CUs and the CSP.
to the cost of the optimal provisioning policy. Compared to The results of the above experiments with PlanetLab
the costs of the Round-Robin provisioning policy, the other cloud tasks are shown in Figure 7 and Figure 8. From
three results average 22% lower. Figure 6) shows that the Figure 7, the total costs of Round-Robin provisioning
highest costs obtained by a single cloud server are very policy are much higher than those of the other three
close to each other among the last three policies, but under policies; in the worst case, the Round-Robin provisioning
the Round-Robin policy, the results can be 2 times those policy result is approximately 2.6 times as large as the
of the other three policies. This result shows that in the optimal provisioning policy result. However, the total costs
Round-Robin provisioning policy, CUs cannot obtain the obtained by Nash provisioning policy and Nash-Stackelberg
resources at a lower price on this cloud server, resulting in provisioning policy are close to the optimal provisioning
an increase in the purchasing cost. This result may occur policy result, and they average 54% below the Round-Robin
because of the pricing differences of heterogeneous cloud provisioning policy costs. Similar to Figure 6, Figure 8
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 7

Fig. 5. Total costs with randomly generated tasks

Fig. 6. Highest income of a single server with randomly generated tasks

shows the highest income from a single cloud server under The experimental results showed multi-dimensional re-
different resource provisioning policies and number of CUs source combinatorial pricing mechanism is more suitable
when running PlanetLab cloud tasks. These results show for describing a real cloud environment executing various
that the proposed provisioning policy based on a multi- types of cloud tasks. For both randomly generated cloud
dimensional pricing mechanism is also applicable to real tasks or actual PlanetLab cloud tasks, CORPP reduced
tasks. the total costs of all CUs efficiently compared to the
existing Round-Robin strategy. Future work should focus
5 C ONCLUSION on strategies that include games between multiple CSPs or
on other heterogeneous cloud data centers objectives, such
In this study, we first designed a multi-dimensional resource
as saving energy.
pricing mechanism for heterogeneous cloud computing sys-
tems that overcomes the drawback of single resource pric-
ing mechanisms, which are unsuitable for the diversity of
cloud computing tasks. Then, we proposed a cost-optimized ACKNOWLEDGMENT
resource provisioning policy (CORP P ) based on game
theory, which considers interactions between the CUs and This work is supported by Doctoral Research Foundation
between the CUs and the CSP to avoid unnecessarily high of Liaoning under grant No. 20170520306, Social Sci-
costs. In these games, CORP P simultaneously considers ence Planning Foundation of Liaoning under grant No.
the existence of Nash equilibrium between the CUs, and L15AGL005 and National Science Foundation of China
Stackelberg equilibrium between the CUs and the CSP. under grant No. 61771090.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 8

Fig. 7. Total costs with PlanetLab tasks

Fig. 8. Highest income of a single cloud server with PlanetLab tasks

R EFERENCES [8] Q. Zhang, L. T. Yang, X. Liu, Z. Chen, and P. Li, ”A Tucker

Deep Computation Model for Mobile Multimedia Feature Learning,”
[1] Q. Zhang, L. T. Yang, Z. Chen, and P. Li, ”PPHOPCM: Privacy- ACM Transactions on Multimedia Computing, Communications, and
preserving High-order Possibilistic c-Means Algorithm for Big Data Applications, vol. 13, no. 3s, pp. 39:1-39:18, 2017.
Clustering with Cloud Computing,” IEEE Transactions on Big Data, [9] T. Roughgarden and E. Tardos, ”How Bad is Selfish Routing?,”
2017, DOI: DOI: 10.1109/TBDATA.2017.2701816. Journal of the ACM, vol. 49, no. 2, pp. 236-259, 2002.
[2] Q. Zhang, L. T. Yang, Z. Chen, P. Li, and M. J. Deen, ”Privacy- [10] T. Roughgarden , ”Stackelberg Scheduling Strategies,” SIAM Journal
preserving Double-projection Deep Computation Model with Crowd- on Computing, vol. 33, no. 2, pp. 332-350, 2004.
sourcing on Cloud for Big Data Feature Learning,” IEEE Internet of [11] R. N. Calheiros, R. Ranjan, A. Beloglazov, A. F. De Rose Cesar,
Things Journal, 2017, DOI: 10.1109/JIOT.2017.2732735. and R. Buyya, ”CloudSim: a Toolkit for Modeling and Simulation of
[3] P. Li, Z. Chen, L. T. Yang, Q. Zhang, and M. J. Deen, ”Deep Cloud Computing Environments and Evaluation of Resource Provi-
Convolutional Computation Model for Feature Learning on Big Data sioning Algorithms,” Software: Practice and Experience, vol. 41, no.
in Internet of Things,” IEEE Transactions on Industrial Informatics, 1, pp. 23-50, 2011.
2017, DOI: 10.1109/TII.2017.2739340. [12] Q. Zhang, L. T. Yang, Z. Chen, and F. Xia, ”A High-order Possibilis-
[4] H. Li, C. Wu, Z. Li, and F. C. M. Lau, ”Profit-maximizing Virtual tic C-means Algorithm for Clustering Incomplete Multimedia Data,”
Machine Trading in a Federation of Selfish Clouds,” in Proceedings of IEEE Systems Journal, vol. 11, no. 4, pp. 2160-2169, 2017.
IEEE International Conference on Computer Communications, 2013, [13] H. Zhou, Victor C. M. Leung, C. Zhu, S. Xu, and J. Fan, ”Predicting
pp. 25-29. Temporal Social Contact Patterns for Data Forwarding in Opportunis-
[5] I. Goiri, J. Guitart, and J. Torres, ”Economic Model of a Cloud tic Mobile Networks,” IEEE Transactions on Vehicular Technology,
Provider Operating in a Federated Cloud,” Information Systems Fron- vol. 66, no. 11, pp. 10372-10383, 2017.
tiers, vol. 14, no. 4, pp. 827-843, 2012.
[6] A. N. Toosi, R. N. Calheiros, R. K. Thulasiram, and R. Buyya,
”Resource Provisioning Policies to Increase IaaS Provider’s Profit in a
Federated Cloud Environment,” in Proceedings of IEEE International
Conference on High Performance Computing and Communications,
2011, pp. 279-287.
[7] M. A. Arfeen, K. Pawlikowski, and A. Willig, ”A Framework for
Resource Allocation Strategies in Cloud Computing Environment.” in
Proceedings of IEEE Computer Software and Applications Conference
Workshops, 2011, pp. 261-266.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 9

Xin Chen received his B.E. degree in com-

puter science from Dalian University of Tech-
nology, China, in 2005 and Ph. D degree
in information science from Poznan Univer-
sity of Technology, Poland, in 2014. He is
currently an associate professor of school
of electronic and information engineering in
Liaoning University of Technology, China. His
research interests include combination opti-
mization, especially scheduling problem and
algorithm design.

Feng Ding received his B.E. degree in Power

Engineering from Dalian University of Tech-
nology, China, in 2002 and Ph. D degree
in Management Science and Engineering
from Dalian University of Technology, China,
in 2009. He is currently an associate pro-
fessor in School of software Technology in
Dalian University of Technology, China. His
research interests include e-government, e-
commerce, decision support, and informa-
tion security.

Tiantian Zhang received her B.E. degree in

software engineering from Dalian University
of Technology, China, in 2017 and she is cur-
rently a master student of School of software
technology in Dalian University of Technol-
ogy, China. Her research interests include
scheduling problem and algorithm design.

Gang Hou received his B.E. degree in com-

puter science from Dalian University of Tech-
nology, China, in 2005 and Ph. D degree in
software engineering from Dalian University
of Technology, China, in 2017. He is currently
an assistant professor of school of software
technology in Dalian University of Technol-
ogy, China. His research interests include
formal methods, model checking and trust-
worthy software.

Lan Lan received her B.E. degree in in-

ternational trade from Victoria University of
Technology, Australia, in 2002 and Master
degree in industrial relations and human re-
source management from University of Syd-
ney, Australia, in 2006. She is currently an
assistant professor of International School
in Shenyang University, China. Her research
interests include cloud computing and re-
source management.

2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.