Pay as How Well You Do: A Quality Based Incentive

Mechanism for Crowdsensing ∗

Dan Peng, Fan Wu†, and Guihai Chen

Shanghai Key Laboratory of Scalable Computing and Systems,
Department of Computer Science and Engineering, Shanghai Jiao Tong University, China
[email protected], {fwu, gchen}@cs.sjtu.edu.cn

ABSTRACT Keywords
In crowdsensing, appropriate rewards are always expected to Crowdsensing; Incentive Mechanism; Quality Estimation;
compensate the participants for their consumptions of phys- Maximum Likelihood Estimation; Information Theory
ical resources and involvements of manual efforts. While
continuous low quality sensing data could do harm to the 1. INTRODUCTION
availability and preciseness of crowdsensing based services,
Crowdsensing is a new paradigm of applications that en-
few existing incentive mechanisms have ever addressed the
ables the ubiquitous mobile devices with enhanced sensing
issue of sensing data’s quality. The design of quality based
capabilities to collect and to share local information toward-
incentive mechanism is motivated by its potential to avoid
s a common goal [5, 12]. In recent years, a wide variety
inefficient sensing and unnecessary rewards. In this paper,
of applications have been developed to realize the poten-
we incorporate the consideration of data quality into the de-
tial of crowdsensing throughout everyday life, such as envi-
sign of incentive mechanism for crowdsensing, and propose
ronmental quality monitoring [2, 3], noise pollution assess-
to pay the participants as how well they do, to motivate
ment [16, 24], road and traffic condition monitoring [19, 31],
the rational participants to perform data sensing efficiently.
road-side parking statistics [18, 21], and indoor localization
This mechanism estimates the quality of sensing data, and
[23,33]. However, the success of crowdsensing based services
offers each participant a reward based on her effective con-
critically depends on sufficient and reliable data contribu-
tribution. We also implement the mechanism and evaluate
tions from individual participants.
the improvements in terms of quality of service and profit
Sensing, processing, and transmitting data in crowdsens-
of service provider. The evaluation results show that our
ing applications requires manual efforts and physical resources.
mechanism achieves superior performance when compared
Therefore, appropriate rewards are always expected to com-
to the uniform pricing scheme.
pensate the owners of task-taking mobile devices. These
owners, or say participants in the literature of crowdsensing,
Categories and Subject Descriptors are commonly assumed to be rational, and would not make
contributions unless there are sufficient incentives. Although
C.2.1 [Computer-Communication Networks]: Network
researchers have proposed a number of incentive mechanisms
Architecture and Design
for participation in crowdsensing [9, 11, 13, 15, 32, 36, 39, 40],
† they have not fully exploited the connection between quality
F. Wu is the corresponding author. of sensing data and rewards for contributions.
∗This work was supported in part by the State Key Devel-
Sensing data of high quality, based on which the crowd-
opment Program for Basic Research of China (973 project
2014CB340303), in part by China NSF grant 61422208, sensing service provider aggregates and extracts informa-
61472252, 61272443, and 61133006, in part by CCF-Intel Y- tion for accurate decision making and attentive service pro-
oung Faculty Researcher Program and CCF-Tencent Open viding, is fundamentally important. In crowdsensing, qual-
Fund, in part by the Scientific Research Foundation for the ity of sensing data can be affected by the characteristics
Returned Overseas Chinese Scholars, and in part by Jiang- of mobile sensors, the clarity of task instructions, as well
su Future Network Research Project No. BY2013095-1-10. as the expertise and willingness of individual participants
The opinions, findings, conclusions, and recommendations
expressed in this paper are those of the authors and do not [14, 20, 25]. Particularly, participants with different spatial-
necessarily reflect the views of the funding agencies or the temporal contexts and personal effort levels are likely to
government. submit sensing data of diverse quality. Furthermore, ra-
tional participants tend to strategically minimize their ef-
forts, while doing the sensing tasks, and thus may degrade
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
the quality of sensing data. For example, careless or indif-
for profit or commercial advantage and that copies bear this notice and the full cita- ferent submissions are always found in crowdsensing based
tion on the first page. Copyrights for components of this work owned by others than noise monitoring applications. When asked for environmen-
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- tal sound heard of neighborhood, a participant may perform
publish, to post on servers or to redistribute to lists, requires prior specific permission the sensing tasks through a mobile device placed inside her
and/or a fee. Request permissions from [email protected].
pocket, rather than carefully taking out the device to sense
MobiHoc’15, June 22–25, 2015, Hangzhou, China.
Copyright c 2015 ACM 978-1-4503-3489-1/15/06 ...$15.00.
accurately. Such a low quality submission would invalidate
http://dx.doi.org/10.1145/2746285.2746306. the estimation of noise pollution.

177
 Continuous low quality sensing data undoubtedly do har- The rest of the paper is organized as follows. We briefly
m to the availability and preciseness of crowdsensing based review related work in Section 2 and present technical pre-
services. However, to the best of our knowledge, few ex- liminaries in Section 3. The detailed design of our quality
isting works have taken the observation of data quality in- based incentive mechanism is presented in Section 4. In Sec-
to consideration, when designing incentive mechanisms for tion 5, we evaluate our incentive mechanism and show the
crowdsensing. It is very challenging to design quality based results. Finally, we conclude this paper in Section 6.
incentive mechanisms for crowdsensing.
Most of all, it is technically difficult to estimate the quality 2. RELATED WORK
of sensing data without any prior knowledge of the sensing
The problem of data quality has been widely studied in
behavior of individual participants or the ground truth of
organizational databases and information systems [7, 29],
targeted contexts. Subsequent quality verification would re-
mainly focusing on quality category, attribute and contex-
quire significant investments in deploying particular infras-
tual pattern from a perspective of data consumer. Sachi-
tructures to do on-site sensing and ground truth collecting,
dananda et al. [27] surveyed building blocks and existing
like Model 831-NMS permanent noise monitoring system [4].
approaches related to quality of information in wireless sen-
Lacking in flexibility and scalability, the deployment of tra-
sor networks.
ditional static sensing infrastructures, in turn, negates the
For the paradigm of crowdsensing, Reddy et al. [26] de-
necessity and benefits of crowdsensing.
veloped a recruitment framework to identify and select well-
Second, it is challenging to design incentive mechanism-
suited participants to achieve high utility within a budget.
s that achieve both individual rationality and profit maxi-
Three stages are introduced, including qualifier, assessment,
mization. Here, individual rationality means that a partici-
and progress review. Though reputation system based on
pant should be rewarded no less than her sensing cost, and
performance has been built, this work focuses on the record
the profit of service provider is the difference between the
of participation likelihood (whether the participant would
value of crowdsensing based services and the total rewards to
take a sensing task when given a chance). In addition, it
participants. Deliberate incentive mechanisms are required
shows little consideration about incentives. On the other
to motivate effective data contributions from rational par-
hand, although various empirical experiments [17, 20, 25, 35]
ticipants, and to maintain a robust, profitable market for
demonstrate that financial and social incentives do have an
crowdsensing service provider.
impact on the performance of participants, such as engage-
Third, it is nontrivial to bridge the gap between quality of
ment, compliance and quality, they fail to generalize an in-
sensing data and rewards for contributions. Participants of
centive model to adaptively guide the participants’ behavior.
crowdsensing, who perform the sensing tasks with heteroge-
There are extensive researches targeting the incentive mech-
neous physical resources and manual efforts, and therefore
anism design for crowdsensing. Lee and Hoh [13] proposed
submit sensing data of diverse quality, may require appro-
a reverse auction based dynamic pricing scheme to moti-
priate rewards according to their contributions. While tra-
vate participants to sell their sensing data with claimed bids.
ditional uniform pricing scheme is unfair, the Pay-as-Bid
Yang et al. [32] considered a platform-centric incentive mod-
pricing method used in most of the auction based incentive
el, where the reward is proportionally shared by participants
mechanisms is somehow troublesome for participants and in-
in a Stackelberg game, and a user-centric incentive model,
dulgent of careless behavior. Both of these existing solutions
where participants in the auction bid for tasks and get paid
ignore the quality issue, and thus are unlikely to encourage
no lower than the submitted bids. Koutsopoulos [11] de-
long-term, effective contributions.
signed an incentive mechanism to determine participation
In this paper, we incorporate the consideration of data
level and payment allocation to minimize platform’s com-
quality into the design of incentive mechanism for crwod-
pensation cost with guaranteed service quality. Zhang et
sensing, and propose to pay the rational participants as how
al. [37] and Zhao et al. [40] suggested online incentive mecha-
well they do, to motivate efficient crowdsensing.
nisms to flexibly recruit participants who appear opportunis-
Our main contributions are listed as follows.
tically in the phenomena of interests. Luo et al. [15] studied
• To the best of our knowledge, we are the first to design an incentive mechanism based on all-pay auction with real-
a quality based incentive mechanism for crowdsensing istic constraints such as information asymmetry, population
that directly motivates individual participants to con- uncertainty, and risk aversion. Kawajiri et al. [10] deployed a
tribute high quality sensing data. crowdsensing based wireless indoor localization system, and
• Second, we extend the well-known Expectation Maxi- steered participants to cover sufficient locations to improve
mization algorithm that combines maximum likelihood the quality of service. There is no skill variance and device
estimation and Bayesian inference to estimate the qual- variance in their system. In general, these existing incentive
ity of sensing data, and further apply the classical In- mechanisms either have not considered the quality of sensing
formation Theory to measure the effective contribu- data or have addressed the incentive concerns and quality
tion of sensing data. Based on the estimated quality issues separately. Moreover, few of them have investigated
and contribution, we can determine fair and proper the method to estimate the quality of sensing data.
rewards to the participants. The proposed incentive In contrast, we systemically consider the participants’ will-
mechanism achieves individual rationality and profit ingness to take a sufficient amount of efforts in crowdsensing,
maximization. and bridge the gap between quality of sensing data and re-
• Finally, we implement the incentive mechanism and wards for contributions, by providing a quality based incen-
extensively evaluate its performance. Our evaluation tive mechanism. The quality estimation method applied in
results show that it achieves superior performance in this paper is originally introduced by Dawid and Skene [6],
terms of quality assurance and profit management, when where the expectation maximization (EM) algorithm is used
compared to the uniform pricing scheme. to obtain maximum likelihood estimates of observers’ error

178
 embedded
sensors

1-1. sensing task

2-1. query
1-2. reserve price
2-2. service 1-3. reward
budget coverage
quality profit 1-4. sensing data

service subscriber service provider participants

Figure 1: A general crowdsensing model.

rates and to infer the true response of patients. Wang and cantly depends on the quality of sensing data, e.g., urban
Ipeirotis [30] applied EM algorithm to estimate the quality noise pollution monitoring, which measures ambient noise
of crowdsourced labeling workers. Zhang et al. [38] proposed pollution based on sensing data collected from mobile de-
to combine spectral methods and EM algorithm to address vices. For each piece of sensing data with an error below
the problem of crowdsourced multi-class labeling with an the specified threshold, the service provider gains a value
optimal convergence rate up to a logarithmic factor. Be- V (e.g., the subscription fee from service subscribers). For
yond quality estimation, we also quantify the contribution simplicity, we assume that V is fixed. The objective of the
of sensing data via information theory, and determine fair service provider is to maximize her own profit, by providing
and proper rewards to participants. services with guaranteed quality, and recruiting participants
with proper rewards. The profit is defined as
3. PRELIMINARIES X
Profit , (V − rk ).
In this section, we present the model of quality based ak ∈W
crowdsensing, and key techniques for quality estimation.
In this paper, we focus on the data quality that is specif-
3.1 Crowdsensing Model ically affected by participants’ effort levels for sensing, and
As illustrated by Figure 1, there are three major compo- aim at designing incentive mechanisms for the service provider
nents in the crowdsensing system, i.e., service subscribers to stimulate high quality sensing and long-term, effective
who request services, a service provider who conducts the contributions.
crowdsensing campaign and provides services, and a crowd
of participants who submit sensing data to support the ser-
vices. The crowdsensing process (the right part) can be dis- 3.2 Quality Estimation via EM
cribed as follows. First, the service provider releases a set T For crowdsensing, e.g., urban noise sensing, it is reason-
of sensing tasks (e.g., noise sensing on campus at 10:00 am) able to calibrate the sensing data, to tolerate the inherent
with an incentive announcement and a quality requirement uncertainty of mobile devices. Here, we divide the read-
(e.g., an error threshold). In the phenomena of interest- ing of sensing data into discrete intervals, and suggest the
s, there is a set A = {a1 , a2 , . . . , an } of participants, with service provider to deliver a certain interval to the service
sensors embedded in their mobile devices. Each participant subscribers, rather than an accurate reading, to mitigate the
ak ∈ A bears a private reserve price/sensing cost ck (i.e., a impact of device variance and device error. The discrete in-
monetary value for her consumptions of physical resources tervals are denoted as a set D = {d1 , d2 , . . . , dm }, where each
and involvements of manual efforts), and thus expects a re- interval spans over a range of decibels, and the granularity of
ward for her contribution. Without sufficient rewards, the interval division can be determined by the tradeoff between
participants may not undertake the sensing tasks. The ser- accuracy and complexity.
vice provider estimates the quality qk of sensing data from Regarding the quality of sensing data as a result of the
each participant ak . By taking the profile of the partici- effort levels, we estimate “effort matrix” ek for each par-
pants’ data quality and sensing costs into consideration, she ticipant ak , and map this effort matrix into a scalar qual-
selects a subset W ⊆ A of participants to perform each sens- ity value through function qk = g(ek ). Here, the effort
ing task, and offers each ak ∈ W a certain amount of reward matrix ek is an m × m matrix, with element ekij ∈ [0, 1],
rk according to her effective contribution. After collecting i = 1, . . . , m, j = 1, . . . , m, indicating the probability that
the sensing data for some tasks, the service provider up- participant ak submits a piece of sensing data in interval
dates quality estimation qk for each ak ∈ W to guide the dj while the true reading is in interval di . Particularly,
next round of recruitment (the right part), and extracts in- {ekii |i = 1, . . . , m} contains the probabilities that partici-
formation to provide services (the left part). pant ak obediently performs outside-pocket sensing for each
We consider a general class of crowdsensing applications, of the m possible Pcases. Furthermore, the conditional prob-
in which the availability and preciseness of services signifi- abilities satisfy j ekij = 1.

179
 We note that, the effort matrix can be measured when we Table 1: Key notations
have ground truth for all spatial-temporal contexts. Howev-
er, for crowdsensing, the true reading, or even the interval, Notation Definition
cannot be ascertained in most cases, making the direct veri- T Set of sensing tasks
fication of data quality and the discernment of effort matrix A Set of participants
challenging. In this paper, we resort to the well-known ex- D Set of discrete noise intervals
pectation maximization (EM) algorithm [6] to estimate each At Set of participants who complete task t ∈ T
participant’s effort matrix. Tk Set of tasks that ak ∈ A performs
The EM algorithm [6] is an iterative method for finding S Set of observed sensing data
the Maximum Likelihood Estimation (MLE) of the parame- P Set of missing true noise interval indicators
ters (e.g., the effort matrix for each participant, and the true E Set of unknown effort matrices
noise interval for each task), when there is missing data (e.g., L(E; P, S) Likelihood function of E
the indicators to tell right or wrong for sensing data) that ek Effort matrix of ak
precludes the straightforward estimation for the parameter- ekij Probability that ak submits data in inter-
s. Here, MLE calculates the best estimation for parameters val dj while the true interval is di
that maximizes the (log-)likelihood of the observations (e.g., Π Noise interval distribution
the submitted sensing data), and converges in probability to pt True noise interval indicator for task t
the true value of the unknown parameters when the number pti Probability of task t with true noise inter-
of observations is sufficiently large. val being di
Given a set S of observed sensing data, a set P of missing dkt Noise interval that ak ’s sensing data for
true interval indicators, a set E of unknown effort matrices, task t falls into
and the density function f , the likelihood of unknown E is I(dkt = dj ) Indicator function for the event dkt = dj
qk Quality of ak ’s sensing data
L(E; P, S) = f (P, S|E).
cm (qk ) Effective contribution of sensing data of
To find the MLE of E, EM algorithm iteratively runs the quality qk
following two steps until convergence (Supposing that E b t is ck Reserve price/sensing cost of ak
the current value of E after t iterations). rk Reward to ak for her contribution
E-step calculates the expected value of likelihood func- V Value gained from qualified sensing data
tion, with respect to the conditional distribution of P given r∗ Optimal quality based reward
observation S under the current estimation of E, ru Optimal uniform reward
bt ) = E
Q(E|E b t [L(E; P, S)].
P |S,E

b that maximizes the ex-

M-step seeks the estimation E the participants’ sensing costs follow a probability distribu-
pectation function, tion, with a probability distribution function f (ck ), and a
cumulative distribution function F (ck ).
b t+1 = arg max Q(E|E
E b t ). A rational participant ak will not do a given sensing task
E unless she gets a reward r ≥ ck . Therefore, the service
Inspired by [6], we extend the algorithm to estimate the provider’s profit by providing services and recruiting partic-
true interval indicators and participants’ effort matrices, by ipant ak , which is defined as the difference between value V
iterating the following two steps until convergence: 1) es- gained from the sensing data, and the reward r to partici-
timate the effort matrix and noise interval distribution via pant ak , where V ≥ r, is formulated as
maximum likelihood estimation, based on the estimated true (
0, r < ck ,
interval indicators; and 2) calculate new estimation of true Profit(ck , r) =
interval indicators, according to the estimated effort matri- V − r, r ≥ ck .
ces and noise interval distribution.
While the distribution of ck is independent of value V and
The converged estimation of participant’s effort matrix
reward r, the expected profit can be calculated as
indicates the quality of sensing data, while the noise interval Z ∞
distribution is suggestive of the noise pollution level.
Profit(r) = Profit(ck , r)f (ck )dck
Z0 r
4. QUALITY BASED INCENTIVE = (V − r)f (ck )dck = F (r)(V − r).
In this section, we detail the design of our quality based in- 0
centive mechanism for crowdsensing. To pay each individual Therefore, the service provider can maximize her profit by
participant ak as how well she does in sensing, we estimate taking the first derivative of the function Profit(r), solving
her effort matrix ek , calculate her quality qk of sensing data, the following equation, and getting the optimal reward,
quantify her effective contribution cm (qk ), and offer a proper
F (r ∗ )
reward rk . Taking the quality of sensing data into consid- r∗ = V − .
eration, our incentive mechanism can encourage long-term, f (r ∗ )
effective contribution for crowdsensing based services.
4.2 Quality Estimation
4.1 A Simple Case In practice, due to their various effort levels, different par-
We first regard all of the submitted sensing data as qual- ticipants may submit sensing data of diverse quality. In this
ified, and present a simple pricing scheme. We assume that subsection, we extend the Estimation Maximization algo-

180
rithm to estimate the effort matrix ek for each participant Algorithm 1: Effort Matrix Estimation
ak , and then estimate the quality of her sensing data as
Input: A set S = {dkt |t ∈ T, ak ∈ At } of observations.
qk = g(ek ). Output: Estimation of effort matrix E, marginal
Specifically, we denote the set of participants that submit distribution of noise interval Π, and posterior
sensing data to task t as At ⊆ A, and the set of tasks that estimation of true noise interval indicators P .
participant ak performs as Tk ⊆ T. For task t ∈ Tk , the // Initialization of True Noise Interval Indicator
true noise interval is denoted as d0t , while the interval into 1 foreach t ∈ T do
which participant ak ’s sensing data falls is denoted as dkt . 2 cnt ← 0; cnt ← 0;
3 foreach ak ∈ At do
An indicator function I(dkt = dj ) (i.e., I(dkt = dj ) = 1 when 4 i ← dkt ; cnti ← cnti + 1; cnt ← cnt + 1;
event dkt = dj is true; otherwise, I(dkt = dj ) = 0) is applied
5 foreach di ∈ D do
to describe the submission of sensing data.
6 pti ← cnti /cnt;
We assume that the effort levels of participants are inde-
pendent, and do not change for a period of time. So that 7 while not converged do
we can periodically learn the effort matrix ek for each par- // Estimation of Effort Matrix
ticipant ak , and put this knowledge into practice. Without 8 foreach ak ∈ A do
the true interval indicator, i.e., pt = {pti |i = 1, . . . , m} for 9 cnt ← 0; ek ← 0;
each task t (pti = 1 if d0t = di for sure) is unavailable, we 10 foreach t ∈ Tk do
11 j ← dkt ;
resort to the EM algorithm that combines Maximum likeli-
12 foreach di ∈ D do
hood estimation and Bayesian inference to iteratively esti- 13 ekij ← ekij + pti ;
mate the unknown effort matrix ek and noise interval dis- 14 cnti ← cnti + pti ;
tribution Π = {πi |i = 1, . . . , m}.
The pseudo-code of this expectation maximization algo- 15 foreach di ∈ D do
rithm is shown in Algorithm 1, which runs as follows. 16 foreach dj ∈ D do
(1) Initialization: For each task t, the probability distri- 17 ekij ← ekij /cnti ;
bution of true noise interval indicator pt is initialized as:
P // Estimation of Noise Interval Distribution
k
ai ∈At I(dt = di )
18 foreach di ∈ D do
pti = p(d0t = di ) = . 19 πi ← 0;
|At | 20 foreach t ∈ T do
(2) Estimation of effort matrix and noise interval distri- 21 πi ← πi + pti ;
bution: Given the likelihood function 22 πi ← πi /|T|;
// Estimation of True Noise Interval Indicator
L(E; P, S) = f (P, S|E), 23 foreach t ∈ T do
and 24 smp ← 0; pt ← 1;
X 25 foreach di ∈ D do
L(E; S) = f (S|E) = f (P, S|E), 26 foreach ak ∈ At do
P 27 j ← dkt ;
28 pti ← pti ekij ;
where E = {ek |ak ∈ A}, P = {pt |t ∈ T}, and S = {dkt |t ∈
29 smp ← smp + πi pti ;
T, ak ∈ At }, the maximum likelihood estimate of E makes
the observation S most likely to happen. 30 foreach di ∈ D do
We note that the effort matrix ek for each participant 31 pti ← πi pti /smp;
ak follows the Multinomial Distribution. When participant
32 Return E = {ek |ak ∈ A},Π = {πi |i = 1, . . . , m},
ak performs nki independent tasks with true interval di , her
P = {pt |t ∈ T};
sensing data for these tasks falls P into interval dj with prob-
ability ekij , where ekij ≥ 0 and j ekij = 1, j = 1, . . . , m. Let
nki1 , . . . , nkim be the number of submissions corresponding
P to
interval d1 , . . . , dm , respectively. Then we have j nkij = nki , interval distribution Π, we apply the Bayesian inference to
and the likelihood function of eki , estimate the true noise interval indicators P . Considering
the n independent observations {S 1 , . . . , S n } of sensing data
nk ! Y k nkij
f (nki1 , . . . , nkim |eki1 , . . . , ekim ) = Q i k (eij ) . from individual participants, where S i = {dit |t ∈ T}, i =
nij ! 1, . . . , m, we have
Taking the log-likelihood, Lagrange multipliers, and deriva- p(P )p(S|P ) p(P )p(S 1 |P ) . . . p(S n |P )
tives, we get the most natural estimates, p(P |S) = = .
p(S) p(S)
P t k
k nkij t∈Tk pi I(dt = dj ) When all terms not involving the true noise interval indica-
ebij = k = P t
, j = 1, . . . , m.
ni t∈Tk pi tor are absorbed into the proportionality sign, we calculate
the distribution of true noise interval indicator according to
The noise interval distribution is estimated as
P Q Q k
t
t∈T pi
πi ak ∈At j (ekij )I(dt =dj )
π
bi = , i = 1, . . . , m. pti = P Q Q k I(dk =d ) , i = 1, . . . , m.
|T| q πq ak ∈At j (eqj )
t j

(3) Estimation of true noise interval indicator: Given the (4) Convergence: We iterate step 2 − 3 until the two esti-
observed sensing data S, the effort matrices E, and the noise b t+1−E
b t |<ε, |Pb t+1−Pbt |<η, ε>0, η>0.
mates converge, i.e., |E

181
For each iteration (the while loop), the computation com- the sensing data can be expressed as mutual information,
plexity is polynomial as O(|A||T||D|).
I(X; Y ) = H(Y ) − H(Y |X)
We claim that the EM algorithm increases the likelihood X
function in each iteration, and finally converges to a stable = H(Y ) − p(x)H(Y |X = x)
estimation. To circumvent the problem of getting trapped x
in a local optimum, we try different initializations for several X
= H(Y ) − p(x)hb (qk )
executions of the algorithm. Although it is hard to provide
x
theoretical guarantee for its performance, the EM algorithm
has been widely used, and a provably optimal convergence = H(Y ) − hb (qk ),
rate up to a logarithmic factor has been shown in [38]. where H(Y ) is entropy of Y , H(Y |X) is the conditional
With the estimation for effort matrix ek , we can get the entropy of Y given X, and hb (qk ) is a binary entropy for the
quality of ak ’s sensing data through the mapping function. binary random noise Z with distribution {qk , 1 − qk }.
For simplicity,
P we focus on pure obedience, and set qk = Intuitively, when no sensing data is submitted, all the m
g(ek ) = i ekii /m. With the estimation for distribution of optional intervals are equally likely to be observed with prob-
true noise interval indicator pt = {pt1 , pt2 , . . . , ptm } for task ability 1/m, making the uncertainty maximal at
t, the interval d∗i to be delivered is the one with maximum X
possibility, i.e., d∗i = arg max pti . H(Y ) = − (1/m) log(1/m) = log(m).
i
m

Given the sensing data, the information uncertainty is re-

4.3 Contribution Quantification duced to be
Various analyses and experiments have confirmed that ex-
pert work can be accomplished by the local crowd, even hb (qk ) = −q log(qk ) − (1 − qk ) log(1 − qk ).
if they are lack of expert knowledge. However, the con- Generally, if Z is not a binary random variable, but dis-
tribution of each individual participant remains unknown. tributed with qk in the correct interval and equal probability
Here, inspired by ideas in Information Theory and Shan- (1 − qk )/(m − 1) for each of the m − 1 rest intervals, then
non’s Channel Coding Theorem [28, 34], we quantify the the information uncertainty is calculated as
participants’ contributions through information uncertain- X
ty reduction. hm (qk ) = −qk log(qk )− ((1 − qk )/(m − 1)) log((1 − qk )/(m − 1)).
We regard the right part of crowdsensing system (Fig- m−1
ure 1) as a signal transmission system (Figure 2), where the
Therefore, the effective contribution of sensing data of qual-
input signal X is the sensing data provided by the partici-
ity qk , can be formulated as
pants, and correspondingly, the output signal Y is the infor-
mation that service provider receives on the sensing data. cm (qk ) = log(m) + qk log(qk ) + (1 − qk ) log((1 − qk )/(m − 1)).
Transmitted through the channel, an input signal may be
distorted in a random way depending on the channel condi- With the convention 0 log 0 = 0, sensing data of quality
tion, and thus the output signal may be different from the qk = 1 will result in minimal uncertainty, hm (1) = 0, and
input signal. Here, the quality qk ∈ [0, 1] of the sensing data maximal contribution, cm (1) = log(m). Though a binary
is expressed by a noise variable Z (independent of X) on the channel which never makes errors and one always makes
transmission channel, where P r(z = 0) = qk indicates that errors are equally good for communication, we only consider
the output signal is equal to the input signal with probabili- and reward sensing data of quality within the range [0.5, 1].
ty qk , and P r(z = 1) = 1 − qk indicates that an error occurs Practically, with the same volume, sensing data of high
with probability 1 − qk . quality carries larger amount of constructive information
than that of low quality. Specifically, the high quality data
contains intrinsic efficiency, while the low quality data need-
quality s extra information, functioning like error-correcting code
restriction (ECC), to detect and/or correct errors without resubmis-
sion. In crowdsensing, such kind of error correction, is more
Z often conducted in the form of verification, such as recruit-
ing another group of participants or sensing another kind of
data (i.e., light signal to determine if the device is out of
pocket). Here, we elide the specific ECC and focus on its
perfect received
cost (i.e., accounting for a part of the data volume), and
sensing data information quantify the effective contribution of sensing data as the in-
X Y formation uncertainty reduction.

Figure 2: A discrete channel (α, Z), where Y = α(X, Z).

4.4 Reward Distribution
In this subsection, we take a step further and reward the
participants proportionally to their quantified contributions,
We assume that, in the signal transmission system, the i.e., r(qk ) = rcm (qk ), where r is a benchmark reward. Not-
input signal is perfect but interfered by the noisy channel ing that we can learn the distribution f (ck , ek ) asymptoti-
with probability 1−qk . Thus, the output signal is equivalent cally, we assume that the distribution is common knowledge.
to received information on sensing data of quality qk . Similar We adjust parameters of the simple case. From partici-
to the capacity of a noisy channel [34], the contribution of pant ak with an effort matrix ek , the profit that the service

182
 80 90 90
Outside Node 1 Node 2 Node 3 Node 1 Node 4 Node 5
Noise Reading (DB) 75 Inside 85 85

Noise Reading (DB)

Noise Reading (DB)

80 80
70
75 75
65 70 70
60 65 65
55 60 60
55 55
50
50 50
45 45 45
40 40 40
0 50 100 150 200 250 300 350 6000 12000 18000 24000 30000 36000 42000 6000 12000 18000 24000 30000 36000 42000
Time Slot Time Slot Time Slot

(a) (b) (c)

90 90 90
Node 1 Node 6 Node 7 Node 1 Node 8 Node 9 Node 1 Node 10 Node 11
85 85 85
Noise Reading (DB)

Noise Reading (DB)

Noise Reading (DB)

80 80 80
75 75 75
70 70 70
65 65 65
60 60 60
55 55 55
50 50 50
45 45 45
40 40 40
6000 12000 18000 24000 30000 36000 42000 6000 12000 18000 24000 30000 36000 42000 6000 12000 18000 24000 30000 36000 42000
Time Slot Time Slot Time Slot

(d) (e) (f)

Figure 3: Accuracy of noise pollution monitoring with different effort levels of participants. (a)General noise reading differences
between outside pocket sensing and inside pocket sensing; (b)-(f) Noise readings of ground truth (Node 1) and from the 10
participants (Node 2-11).

90 90 90
Ground Truth QM Ground Truth MV Ground Truth AA
85 85 85
Noise Reading (DB)

Noise Reading (DB)

Noise Reading (DB)

80 80 80
75 75 75
70 70 70
65 65 65
60 60 60
55 55 55
50 50 50
45 45 45
40 40 40
6000 12000 18000 24000 30000 36000 42000 6000 12000 18000 24000 30000 36000 42000 6000 12000 18000 24000 30000 36000 42000
Time Slot Time Slot Time Slot

(a) Quality Measured Model (b) Majority Voting Model (c) All and Average Model

Figure 4: Comparison of monitoring accuracy of different models.

provider gains from the sensing data is 5.1 Quality Assurance

( We install NoiseTube mobile app [1] on Google Nexus 7,
k
0, rcm (g(ek )) < ck ,
Profit(ck , e , r)= and use the embedded acoustic sensor to measure noise in
V − rcm (g(ek )), rcm (g(ek )) ≥ ck . a meeting room. We recruit 10 participants to take part in
the experiment, each of which carries a nexus and randomly
Then, the optimal quality based reward is determined by
puts it into his/her pocket or on the table. The participants
r∗ = arg max Profit(r) are well told that accurate monitoring occurs when they put
r
Z Z ∞ out the nexuses and keep them undisturbed.
= arg max Profit(ck , ek , r)f (ck , ek )dck dek . The basic experiment is to test whether the participants’
r ek 0 effort levels will effect the noise readings. As Figure 3(a)
shows, the noise reading from a muffled microphone inside
For simple joint distribution f (ck , ek ) of sensing cost and
pocket is at least 5dBs lower than that of outside pocket
effort matrix, the optimal reward r∗ can be calculated by
sensing. The sensing data submitted by participants, as
solving the integral equation and taking the derivation of r.
shown in Figure 3(b)-Figure 3(f), also presents such reading
However, for complex cases, greedy algorithms can find the
differences, based on which we can roughly tell the effort
proper reward with approximate profit more efficiently.
levels of participants, i.e., node 10 is sensing with the highest
effort level and submits almost perfect readings; node 7,
5. EVALUATION RESULTS node 8, node 9 and node 11 are 85% accurate with high effort
levels at most of the time; node 2 and node 4 are helpful
In this section, we conduct simulations to evaluate perfor-
with 70% accuracy; node 6 is careless with high accuracy at
mance of our quality based incentive mechanism. We first
first and then gradually slacks off; node 5 is indifferent with
analyze the improvement in quality assurance. Then, we
half accuracy and the other half deviation intermittently;
compare our quality based reward mechanism to the unifor-
and node 3 is sensing with the lowest effort level with all
m pricing scheme, and illustrate the superior performance
readings lower than ground truth.
in profit management.

183
 80 ground truth), and more robustly to the efforts fluctuation
of participants, especially when careless and indifferent par-
60
ticipants take up more than half of the whole population.
40 Furthermore, the MV model may direct the monitoring into
a fierce fluctuation when the noise interval is highly precise,
20 which is 5dBs per interval in our setting. Despite a similar
trend with QM, the AA model is more vulnerable to large
0
amount of low quality submissions.

4
5.2 Profit Management
To test the performance of our quality based incentive
Reserve Price

3
mechanism in terms of profit management, we first generate
2 the sensing costs and effort matrices for participants, and
then compare the profit of our mechanism to that of the
1 uniform pricing scheme.
We draw vc and ve from a bivariate normal distribution,
0 (c, e) ∼ N (µ1 , µ2 , σ12 , σ22 , ρ), where µ1 = 2.0, µ2 = 0.75,
σ1 = 1.0, σ2 = 0.125, and ρ = 0.0 is set to indicate that
0.5 0.6 0.7 0.8 0.9 1.0 there is no correlation between sensing cost and effort ma-
Effort Level (eii) trix (Figure 5(a)), or ρ = 0.8 for a strong positive correla-
tion (Figure 5(b)). According to the 68 − 95 − 99.7 rule/3σ
(a) No correlation, ρ = 0.0 rule [22], the 95.45% confidence interval is µ ± 2σ, which
empirically states that about 95.45% data drawn from the
80
normal distribution lies within [0.0, 4.0] × [0.5, 1.0] in our
60 setting. Then, we transform vc and ve to ck and ek cor-
respondingly by setting ck = max(−0.5, min(vc , 4.5)) and
40
ekii = max(0.45, min(ve , 1.05)), i = 1, . . . , m. Therefore,
20 the extreme data is excluded and the rest majority approxi-
mately follows the same normal distribution. Notably, other
0 forms of distribution are also experimentally possible, and
the exact joint distribution needs to be carefully estimated
in practical crowdsensing markets [8].
4
After getting the joint distribution, we compare the profit
of our quality based incentive mechanism and the uniform
Reserve Price

3
pricing scheme, where the profit is defined as the difference
2
between the total value gained from the sensing data and the
rewards for contributions. We note that, the quality based
1
incentive mechanism gains a full value V from sensing data
by providing an error-bounded service, and offers each par-
0 ticipant a proper reward based on her effective contribution.
In the uniform pricing scheme, the sensing data is regard-
0.5 0.6 0.7 0.8 0.9 1.0 ed equally with the same quality, and the participants are
Effort Level (eii) offered the same reward,

(b) Positive correlation, ρ = 0.8 ru = max ck .

ak ∈At

However, the gained value is restricted by the actual quality

Figure 5: Joint distribution of sensing cost and effort matrix. qk of sensing data, i.e., vk = y(V, qk ), which monotonously
increases with quality qk .
For simplicity, we consider that there are |D| = 2 noise
Given the reading differences, we compare the quality as- intervals, and omit the subscript of e. Then, the effective
surance, i.e., the overall monitoring accuracy, as a collective contribution is calculated as
work from the crowd, in our quality measured model (QM),
cm (g(e)) = c2 (e) = 1 + e log e + (1 − e) log(1 − e).
traditional majority voting model (MV), and all and aver-
age model (AA). The difference is: QM excludes sensing da- Value function is set to be vk = V sin(cm (g(e))×π/2), which
ta with low quality (i.e., with accuracy less than 50%) and is concave with feasible cm ∈ [0, 1].
assigns quality-estimated data with different weights; MV We select participants from sufficient crowd, in an increas-
selects the most frequent noise interval at first, and then ing order of cost/contribution ratio, and calculate the opti-
calculates the noise reading averagely; and AA takes in all mal reward for the top proportion of them, ranging from
submissions and figures out the average reading. 10% to 100%. The optimal reward, in our quality based
Results, as shown in Figure 4, indicate that QM outper- incentive mechanism, is determined by
forms the other two models, in monitoring the noise pollu-
tion more accurately (i.e., the readings keep closely to the r∗ = arg min rcm (g(ek )) − ck ≥ 0, ∀ak ∈ At .
r

184
Each participant ak will get a proper reward, tween quality of sensing data and proper reward for con-
∗ k tribution, and proposed the quality based incentive mecha-
rk = r cm (g(e )). nism, which achieves both individual rationality and profit
Results, as shown in Figure 6, indicate that our quality maximization. Our incentive mechanism estimates the ef-
based incentive mechanism overwhelmingly outperforms the fort matrix for each participant, calculates the quality of
uniform pricing scheme, in both of the two distributions. sensing data, and offers a reward in accordance with each
The quality based incentive mechanism complies with the effective contribution, aiming to motivate individual partic-
cost/contribution ratio to set the optimal reward in every ipants with different sensing costs to place sufficient manual
stage, and thus can fully leverage the power of participants efforts and submit high quality sensing data in crowdsens-
to complete the sensing tasks at a low cost, when compared ing. We have also implemented part of the mechanism with
to the uniform pricing scheme. Moreover, with the guaran- extensive experiments and simulations. Compared to the
teed value of service, the quality based incentive mechanism, existing uniform pricing scheme, our mechanism achieves
with higher accuracy and less fluctuation in noise monitor- superior performance in profit management.
ing, is more appealing to the service provider.
The results also suggest the proper fraction of partici- 7. ACKNOWLEDGEMENTS
pants that the service provider should try to recruit, which
We appreciate the anonymous reviewers, whose comments
is 80% for both schemes when sensing cost and effort ma-
led to an improvement of this paper. Our shepherd, Vishal
trix has no correlation, and 80% and 70%, for our quality
Misra, gave us highly valuable comments to improve it.
based incentive mechanism and the uniform pricing scheme,
respectively, when the factors are positively correlated. It
is reasonable to see such a turning point in profit, from a 8. REFERENCES
smooth rise to a fall, since there are always some partici- [1] Noisetube. http://www.noisetube.net/, 2008. [Online].
pants with sensing costs higher than what they deserve to [2] Creek watch. http://creekwatch.researchlabs.ibm.com/,
2010. [Online].
be rewarded according to the their effective contributions,
[3] Opensense. http://www.opensense.ethz.ch/trac/, 2010.
i.e., with unreasonably high sensing costs for subpar con- [Online].
tributions. The x-axis ends with 970 (Figure 6(a)) and 940 [4] Noisemonitoring. http://www.larsondavis.com/, 2015.
(Figure 6(b)), respectively, since we have excluded extreme [Online].
data from the distribution. [5] J. A. Burke, D. Estrin, M. Hansen, A. Parker,
N. Ramanathan, S. Reddy, and M. B. Srivastava.
Participatory sensing. Center for Embedded Network
Quality-based Pricing
2500 Uniform Pricing Sensing, 2006.
[6] A. P. Dawid and A. M. Skene. Maximum likelihood
2000
estimation of observer error-rates using the EM algorithm.
Applied Statistics, pages 20–28, 1979.
Profit

1500
[7] D. Hand. Principles of data mining. Drug Safety,
1000 30(7):621–622, 2007.
[8] J. J. Horton and L. B. Chilton. The labor economics of
500 paid crowdsourcing. In Proceedings of the 11th ACM
0
Conference on Electronic Commerce (EC), 2010.
100 200 300 400 500 600 700 800 900 970 [9] L. G. Jaimes, I. Vergara-Laurens, and M. A. Labrador. A
Number of participants location-based incentive mechanism for participatory
(a) No correlation, ρ = 0.0 sensing systems with budget constraints. In Proceedings of
the 2012 IEEE International Conference on Pervasive
Computing and Communications (PerCom), 2012.
Quality-based Pricing
2500 Uniform Pricing [10] R. Kawajiri, M. Shimosaka, and H. Kashima. Steered
crowdsensing incentive design towards quality-oriented
2000 place-centric crowdsensing. In Proceedings of the 16th ACM
International Conference on Ubiquitous Computing
Profit

1500
(UbiComp), 2014.
1000 [11] I. Koutsopoulos. Optimal incentive-driven design of
participatory sensing systems. In Proceedings of the 32nd
500 Annual IEEE International Conference on Computer
Communications (INFOCOM), 2013.
0
100 200 300 400 500 600 700 800 900 940 [12] N. D. Lane, E. Miluzzo, H. Lu, D. Peebles, T. Choudhury,
Number of participants and A. T. Campbell. A survey of mobile phone sensing.
IEEE Communications Magazine, 48(9):140–150, 2010.
(b) Positive correlation, ρ = 0.8
[13] J.-S. Lee and B. Hoh. Sell your experiences: a market
mechanism based incentive for participatory sensing. In
Figure 6: Comparison on profit of different pricing schemes. Proceedings of the 2010 IEEE International Conference on
Pervasive Computing and Communications (PerCom),
2010.
[14] H. Lu, W. Pan, N. D. Lane, T. Choudhury, and A. T.
6. CONCLUSION Campbell. Soundsense: Scalable sound sensing for
people-centric applications on mobile phones. In Proceeding
In this paper, we have incorporated the consideration of of the 7th Annual International Conference on Mobile
data quality into the design of incentive mechanism for crowd- Systems, Applications, and Services (MobiSys), 2009.
sensing. By applying the expectation maximization algo- [15] T. Luo, H.-P. Tan, and L. Xia. Profit-maximizing incentive
rithm and information theory, we have bridged the gap be- for participatory sensing. In Proceedings of the 33rd Annual

185
 IEEE International Conference on Computer [28] C. E. Shannon. A mathematical theory of communication.
Communications (INFOCOM), 2014. The Bell System Technical Journal, 27:379–423,623–656,
[16] N. Maisonneuve, M. Stevens, M. Niessen, and L. Steels. 1948.
Noisetube: Measuring and mapping noise pollution with [29] D. M. Strong, Y. W. Lee, and R. Y. Wang. Data quality in
mobile phones. In Information Technologies in context. Communication of the ACM, 40(5):103–110, 1997.
Environmental Engineering. 2009. [30] J. Wang and P. G. Ipeirotis. Quality-based pricing for
[17] W. Mason and D. J. Watts. Financial incentives and the crowdsourced workers. In Technical Report, New York
performance of crowds. SIGKDD Explorations Newsletter, University, 2013.
11(2):100–108, 2010. [31] Y. Wang, X. Liu, H. Wei, G. Forman, C. Chen, and
[18] S. Mathur, T. Jin, N. Kasturirangan, J. Chandrasekaran, Y. Zhu. Crowdatlas: Self-updating maps for cloud and
W. Xue, M. Gruteser, and W. Trappe. Parknet: Drive-by personal use. In Proceeding of the 11th Annual
sensing of road-side parking statistics. In Proceeding of the International Conference on Mobile Systems, Applications,
8th Annual International Conference on Mobile Systems, and Services (MobiSys), 2013.
Applications, and Services (MobiSys), 2010. [32] D. Yang, G. Xue, X. Fang, and J. Tang. Crowdsourcing to
[19] P. Mohan, V. N. Padmanabhan, and R. Ramjee. Nericell: smartphones: incentive mechanism design for mobile phone
rich monitoring of road and traffic conditions using mobile sensing. In Proceeding of the 18th Annual International
smartphones. In Proceedings of the 6th ACM Conference Conference on Mobile Computing and Networking
on Embedded Networked Sensor Systems (SenSys), 2008. (MobiCom), 2012.
[20] M. Musthag, A. Raij, D. Ganesan, S. Kumar, and [33] Z. Yang, C. Wu, and Y. Liu. Locating in fingerprint space:
S. Shiffman. Exploring micro-incentive strategies for wireless indoor localization with little human intervention.
participant compensation in high-burden studies. In In Proceeding of the 18th Annual International Conference
Proceedings of the 13th ACM International Conference on on Mobile Computing and Networking (MobiCom), 2012.
Ubiquitous Computing (UbiComp), 2011. [34] R. W. Yeung. Information Theory and Network Coding.
[21] S. Nawaz, C. Efstratiou, and C. Mascolo. Parksense: a Springer, 2008.
smartphone based sensing system for on-street parking. In [35] L. Yu, P. André, A. Kittur, and R. Kraut. A comparison of
Proceeding of the 19th Annual International Conference on social, learning, and financial strategies on crowd
Mobile Computing and Networking (MobiCom), 2013. engagement and output quality. In Proceedings of the 17th
[22] F. Pukelsheim. The three sigma rule. American ACM Conference on Computer Supported Cooperative
Statistician, 44:88–91, 1994. Work and Social Computing (CSCW), 2014.
[23] A. Rai, K. K. Chintalapudi, V. N. Padmanabhan, and [36] Q. Zhang, Y. Wen, X. Tian, X. Gan, and X. Wang.
R. Sen. Zee: zero-effort crowdsourcing for indoor Incentivize crowd labeling under budget constraint. In
localization. In Proceeding of the 18th Annual International Proceedings of the 34th IEEE International Conference on
Conference on Mobile Computing and Networking Computer Communications (INFOCOM), 2015.
(MobiCom), 2012. [37] X. Zhang, Z. Yang, Z. Zhou, H. Cai, L. Chen, and X. Li.
[24] R. K. Rana, C. T. Chou, S. S. Kanhere, N. Bulusu, and Free market of crowdsourcing: Incentive mechanism design
W. Hu. Ear-phone: An end-to-end participatory urban for mobile sensing. IEEE Transactions on Parallel and
noise mapping system. In Proceedings of the 9th Distributed Systems, 25(12):3190–3200, 2014.
ACM/IEEE International Conference on Information [38] Y. Zhang, X. Chen, D. Zhou, and M. I. Jordan. Spectral
Processing in Sensor Networks (IPSN), 2010. methods meet EM: A provably optimal algorithm for
[25] S. Reddy, D. Estrin, M. Hansen, and M. Srivastava. crowdsourcing. In Proceedings of the 28th Annual
Examining micro-payments for participatory sensing data Conference on Neural Information Processing Systems
collections. In Proceedings of the 12th ACM International (NIPS), 2014.
Conference on Ubiquitous Computing (UbiComp), 2010. [39] Y. Zhang and M. van der Schaar. Reputation-based
[26] S. Reddy, D. Estrin, and M. Srivastava. Recruitment incentive protocols in crowdsourcing applications. In
framework for participatory sensing data collections. In Proceedings of the 31st Annual IEEE International
Proceedings of the 2010 IEEE International Conference on Conference on Computer Communications (INFOCOM),
Pervasive Computing and Communications (PerCom), 2012.
2010. [40] D. Zhao, X.-Y. Li, and H. Ma. How to crowdsource tasks
[27] V. Sachidananda, A. Khelil, and N. Suri. Quality of truthfully without sacrificing utility: Online incentive
information in wireless sensor networks: A survey. In mechanisms with budget constraint. In Proceedings of the
Proceeding of the 12th Annual International Conference on 33rd Annual IEEE International Conference on Computer
Information Quality (ICIQ), 2010. Communications (INFOCOM), 2014.

186