Judgment and Decision Making, Vol. 2, No. 2, April 2007, pp. 96106.

An alternative approach for eliciting willingness-to-pay: A

randomized Internet trial
Laura J. Damschroder
1,3
, Peter A. Ubel
1,2,3,4
, Jason Riis
5
, and Dylan M. Smith
1,2,3
1
HSR&D Ann Arbor Center of Excellence, Department of Veterans Affairs, Ann Arbor, MI
2
Division of General Internal Medicine, University of Michigan
3
The Center for Behavioral and Decision Sciences in Medicine, University of Michigan
4
Department of Psychology, University of Michigan
5
Department of Marketing, Stern School of Business, New York University
Abstract
Open-ended methods that elicit willingness-to-pay (WTP) in terms of absolute dollars often result in high rates of
questionable and highly skewed responses, insensitivity to changes in health state, and raise an ethical issue related to
its association with personal income. We conducted a 2x2 randomized trial over the Internet to test 4 WTP formats: 1)
WTP in dollars; 2) WTP as a percentage of nancial resources; 3) WTP in terms of monthly payments; and 4) WTP as
a single lump-sum amount. WTP as a percentage of nancial resources generated fewer questionable values, had better
distribution properties, greater sensitivity to severity of health states, and was not associated with income. WTP elicited
on a monthly basis also showed promise.
Keywords: health, contingent valuation, willingness-to-pay, computerized elicitation, income.
1 Introduction
Many economists elicit peoples willingness to pay
(WTP) for healthcare interventions through contingent
valuation surveys so that the benets of those interven-
tions can be valued in monetary terms (Diener, OBrien,
& Gafni, 1998; Klose, 1999; Olsen & Smith, 2001;
Smith, 2003). This is despite many known biases that
occur when attempting to elicit a dollar value from peo-
ple for a good that is not usually directly available in the
market; e.g., perfect health (Baron, 1997). Much litera-
ture focuses on developing consensus on the most valid
method for eliciting WTP; putting aside any philosoph-

The authors would like thank Richard Smith for his insightful com-
ments on earlier drafts of this paper. Also, thanks to Todd Roberts and
Jennifer Heckendorn who helped administer and implement the survey.
Financial disclosure: This research was supported by HSR&D Ann
Arbor Center of Excellence, Department of Veterans Affairs and the
National Institute on Child Health and Human Development Grant
#R01HD040789. The funding agreement ensured the authors inde-
pendence in designing the study, interpreting the data, writing and pub-
lishing the report. The following authors are employed by the VA Ann
Arbor Healthcare System: Laura J. Damschroder, Dylan Smith, and Pe-
ter A. Ubel. Dylan Smith is supported by a career development award
from the Department of Veterans Affairs.
Direct Correspondence to: Laura J. Damschroder, University of Michi-
gan Health System, 300 North Ingalls, Room 7C27, Ann Arbor, MI
481090429. Email: [email protected]
ical issues that question the validity of eliciting WTP
through a single elicitation. Early WTP surveys elicited
values using an open-ended question from a self-interest
perspective to obtain personal use values; e.g. howmuch
would you be willing to pay to be cured? (Smith &
Richardson, 2005). These open-ended formats ask for
WTP values without presenting a starting point value and
without using a search routine to help respondents de-
termine a value. Respondents are simply asked to give
a dollar value. However, researchers have questioned
the validity of this format because responses are prone
to a high number of non-response or zero values and
because responses are heavily skewed toward high val-
ues, perhaps, in part, due to strategic bias (Donaldson,
Thomas, & Torgerson, 1997; OBrien & Gafni, 1996). In
response to these concerns, a U.S. Federal panel in 1993,
led by Kenneth Arrow, concluded that both experience
and logic suggest that responses to open-ended questions
will be erratic and biased (Arrow et al., 1993, p. 4613).
Since then, researchers have moved away from elic-
iting WTP using an open-ended format and developed
three types of closed-ended formats in an attempt to over-
come shortcomings of the open-ended format. These
close-ended formats ask respondents to say yes or no
to a series of questions or to select a value from a pre-
specied list. All three methods have methodological is-
96
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 97
sues, however. The bidding game is prone to starting-
point bias (WTP changes depending on the starting value
used to begin the bidding) and the payment card method
is prone to range bias (WTP changes depending on the
range of values presented) (Klose, 1999; Smith, 2000;
Venkatachalam, 2004; Whynes, Wolstenholme, & Frew,
2004). The single-bounded discrete choice format is sta-
tistically inefcient and studies using this approach are
very expensive to conduct because, all else being equal,
it requires a larger sample size and more sophisticated
design and analysis techniques (Smith, 2000; Venkat-
achalam, 2004). In addition, this format is prone to
several biases including yea-saying where respondents
have a tendency to agree with the amount presented (Ye-
ung, Smith, Ho, Johnston, & Leung, 2006). A double-
bounded choice format was derived to increase statistical
efciency. However, even responses from people who re-
port a high level of certainty about their willingness to
pay exhibit signicant anomalies that increase as uncer-
tainty increases (Watson & Ryan, 2006).
We believe the open-ended format deserves further ex-
ploration. Despite the strong statement we quoted ear-
lier against using it, some researchers do not agree with
the call to abandon the open-ended format (Smith, 2000).
Although different formats produce different responses,
it is not clear which format is superior (Venkatachalam,
2004). A recent study comparing alternate elicitation for-
mats concluded, . . . it would seem that the most informa-
tive elicitation format in the present context . . . appear[s]
to be the open-ended format. . . [though this] format is
nowadays distinctly unfashionable in health economics,
having long since given way to supposedly-superior elic-
itation formats (Whynes, Frew, & Wolstenholme, 2005,
p. 384). Advantages of the open-ended format are that
it does not introduce range or starting-point biases and it
can be highly statistically efcient compared to discrete
choice formats.
The open-ended format also has several clear disadvan-
tages, however. This format may place a heavy cognitive
demand on respondents. In fact, the other formats were
developed, in part, to make the elicitation simpler and
more realistic for respondents (Donaldson et al., 1997;
Smith, 2000). Furthermore, asking for WTP in terms
of dollars using an open-ended format requires using an
unbounded response scale (a scale that starts at zero but
with no dened upper end) that naturally contributes to
the highly variable and skewed responses typically seen
with open-ended WTP elicitations (Kahneman, Ritov, &
Schkade, 1999). In addition, people may be more likely
to give strategic values with an unbounded scale; a re-
spondent may believe that the treatment has high intrin-
sic or social value and thus places a very high value not
grounded in the reality of actually paying such a gure in
the form of taxes or as an out-of-pocket expense (Arrow
et al., 1993). Conversely, a respondent may give an arti-
cially low response in an attempt to inuence the actual
price eventually charged.
It could be that a more constrained, but still essentially
open-ended approach might avoid some of the problems
reviewed above. Specically, eliciting WTP as a per-
centage of nancial resources has two potential advan-
tages. First, a percentage measure will force the use of
a bounded 0100 response scale creating a more statis-
tically efcient scale measure (Kahneman et al., 1999).
Generally, people are unable to map their preference for
a health effect using a scale consisting of dollars with that
starts at zero but with no clear maximum amount (an un-
bounded scale) (Payne, Bettman, & Schkade, 1999). Sec-
ond, percentages involve smaller numbers (a 0100 scale
for the percentage formats versus 0 to an undened max-
imum for the dollar formats) and people process smaller
whole numbers more reliably. In one study, Thompson,
Read, and Liang (1984) found that a percentage measure
exhibited more signicant associations with key indepen-
dent variables such as the number of symptoms suffered
by respondents and medications taken than did WTP ex-
pressed in dollars.
The purpose of the current study was to compare WTP
values elicited as a percentage of nancial resources to
values elicited as dollars using open-ended formats. We
predicted that the percentage method would be less prone
to inconsistent responses, would be more sensitive to dif-
ferences in severity across health states, and would show
more desirable distributional properties. We asked for
percentages based on nancial resources rather than in-
come because it is realistic to expect that many people
would consider savings, borrowing power, and other -
nancial resources to pay for a cure of a condition they
want to avoid. Thinking about paying out amounts on a
monthly basis rather than a single lump sum enables re-
spondents to think of smaller quantities and the amounts
proposed are likely to be more salient because many peo-
ple budget their nances on a monthly basis. Advantages
of the percentage format could be reduced or eliminated
when monthly payments rather than lump sum payments
are considered. Thus, we also introduced a second di-
mension against which to compare elicitation formats: a
monthly timeframe versus a single lump sum amount.
The current study extends the studies done by Thomp-
son and colleagues (the largest study, to date, that has
elicited WTP as a percentage) in several ways. First, we
introduce a within-subjects measure of sensitivity. Sec-
ond we compare the effects of using a monthly timeframe
to elicit WTP to a single lump-sum amount. Third, we fo-
cus specically on distributional properties of responses
to further assess percentage formats as a more efcient
measure. Finally, the current study utilizes a larger sam-
ple, and surveys the general public instead of patients.
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 98
Table 1: WTP elicitation formats.
Time Period:
WTP Units: Total Monthly
$US Please type the maximum dollar
amount you think you would be will-
ing and able to pay for this treatment.
$_____ (please enter only one amount)
Please type the maximum dollar
amount you think you would be
willing and able to pay per month for
this treatment.
$_____ per month. (please enter
only one amount)
% nancial resources Please type the maximum percentage
of your nancial resources you think
you would be willing and able to pay
for this treatment.
_____ % of my nancial resources.
(please enter a number between 0 and
100)
Please type the maximum percentage
of your nancial resources you think
you would be willing and able to pay
per month for this treatment.
_____% of my nancial resources
per month. (please enter a number
between 0 and 100)
2 Method
We elicited peoples WTP for curing two health condi-
tions using a web-based survey over the Internet. We
recruited respondents via an email sent to a sample of
members in an Internet panel maintained by Survey Sam-
pling International (SSI). This panel is made up of more
than 1 million unique member households, recruited via
random digit dialing, banner ads, and other opt-in tech-
niques. Our study sample was stratied to mirror the U.S.
census population based on age, gender, race, education
level, and income. Upon completion of the survey, par-
ticipants were entered into a drawing for cash prizes that
totaled $10,000.
2.1 Health state descriptions
We presented descriptions of two health states to each
respondent: 1) a below-the-knee amputation (BKA) that
moderately affects physical mobility; and 2) paraplegia,
which signicantly affects mobility. Detailed health state
descriptions are in the appendix. We counterbalanced the
order of the BKA and paraplegia health states.
2.2 WTP elicitation formats
We elicited each respondents WTP for a medical treat-
ment that would permanently restore full physical func-
tioning for each of the two health states. Respondents
were randomly assigned to one of four elicitation for-
mats, using a full-factorial two-by-two experimental de-
sign. We elicited WTP using one of two different units
of measure (percentage of nancial resources or dollars)
and one of two different timeframes (on a monthly ba-
sis or an overall total). No durations for payments were
specied. We chose percentage of nancial resources
instead of income for reasons already cited. Financial re-
sources will typically be equal to or greater than income;
thus, the underlying scale could represent values greater
than income. The four versions (2 WTP measures X 2
timeframes), along with the specic questions we posed
are presented in Table 1.
For each format, we rst presented the description of
the health state (listed in the appendix) and then asked the
respondent to type in their response. The precise word-
ing asking for a WTP amount depended on the format to
which the respondent was assigned, as presented in Table
1. We then told respondents, In answering this question,
take into consideration the actual nancial resources you
have. We recognize that giving an exact amount may be
difcult; just give the best estimate you can. Our purpose
with this instruction was to emphasize personal nancial
constraints before respondents gave a WTP amount. We
elicited WTP for both health states from each respondent.
2.3 Outcome criteria and analysis ap-
proach
Analyses were performed using the native units and time-
frame with which WTP was elicited; e.g., in terms of
monthly percentage of nancial resources. Our primary
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 99
study question was whether WTP expressed as a percent-
age of nancial resources would result in higher quality
responses and better distributional properties compared
to WTP expressed in absolute dollars, and thus would
show greater ability to detect differences between health
states of different severity. We also wanted to explore
whether WTP expressed on a monthly basis would im-
prove properties of WTP responses and perhaps reduce
any advantages observed of the percentage format.
We compared the four elicitation formats using ve cri-
teria:
First, we wanted to reduce the number of questionable
WTP responses. Questionable WTP responses include
missing values, values of zero, or WTP values that are
the same for both health states. We used
2
tests to com-
pare differences in frequencies for these types of occur-
rences between the formats. Those who gave missing or
zero values for both health states were excluded from the
remaining analyses.
Second, we assessed normality of WTP values in terms
of skewness and kurtosis. Parametric models are often
used to predict WTP responses and assume that WTP val-
ues and error terms are normally distributed. Even a small
misspecication of the functional form in these analyses
can result in large differences in predictions (Yeung et al.,
2006).
Third, we assessed internal consistency with a simple
ordinal consistency check. WTP values should reect
the lower impact that BKA has on mobility compared to
paraplegia. Accordingly, we expect respondents WTP
for treatments to be lower for BKA compared to paraple-
gia. We excluded cases where the value was the same for
both health states from this portion of the analysis and
they were not included in the denominator. We used
2
tests to compare differences in the proportion of those
who were ordinally consistent between the groups.
Fourth, we tested the sensitivity of each of the WTP
elicitation versions for detecting differences between the
two health states by computing Cohens d-statistic as
a measure of effect size (Cohen, 1988). Larger effect
sizes indicate greater sensitivity and thus will require
smaller samples to detect statistical differences between
two health states.
Our nal assessment was investigating the degree to
which WTP values correlate with reported income for
each of the four formats, using the Spearman rank cor-
relation coefcient. Condence intervals were computed
using the bias-correction and accelerated bootstrap esti-
mation method (Haukoos & Lewis, 2005). Two smaller-
scale studies that elicited WTP as a percentage of wealth
did not nd this measure to be signicantly associated
with personal income (Schiffner et al., 2003; Thomp-
son, 1986). Nonetheless, it is possible that an association
would still persist in our study because people with low
incomes may have fewer discretionary nances available,
even when expressed as a percentage (Donaldson, Birch,
& Gafni, 2002). Though we did not have a prediction
about whether WTP and income would be signicantly
associated with the WTP elicited using the two percent-
age formats, we did hypothesize that WTP as a percent-
age of wealth would have a lower association with in-
come compared to WTP elicited as dollars.
3 Results
Compared to WTP expressed as absolute dollars, WTP
expressed as a percentage of nancial resources gener-
ates more usable values, greater sensitivity to differences
in severity between health states, better distribution prop-
erties, and is not associated with income. Furthermore,
asking WTP in terms of monthly amounts also shows
promise.
3.1 Respondents
Eight percent of those invited responded by clicking onto
our survey using a link from within the email invitation.
Of those who clicked onto the site, 75% (n=982) com-
pleted the survey. Of those who completed the survey,
98% were included in the analyses, except where noted.
5 were excluded because they were under 18 years old,
15 said they intentionally gave wrong answers, and one
gave invalid values (38,117 for both health states using
the monthly percentage format). The rate of exclusions
were similar across the four versions of survey (p=.22.).
The remaining 961 respondents gave 1,812 non-zero and
non-missing WTP valuations; 55 (6%) gave missing or
zero WTP values for both health states.
The 961 respondents included in the analyses were not
statistically different across the experimental groups with
respect to demographic factors (p-values > 0.15). Over-
all, 31% of respondents identied themselves as being a
non-white race or Hispanic ethnicity. Self-reported mean
age was 46 years (s.d.=16). Median education was some
college but no degree. Overall, 59% of respondents were
women. Just under half (44%) of respondents identied
themselves as having average economic status and 47%
of respondents reported an income of $40,000 or less.
3.2 Questionable Values
55 (6%) respondents gave zero or missing values for both
health states. Another 39 (4%) gave a zero or missing
value for one health state. The rate of zero or missing
values was comparable across the four versions (Chi-
square; p=.60). However, the rate of those who gave
zero or missing values for both health states varied by
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 100
Table 2: Summary of outcome criteria.
% Total % Monthly $ Total $ Monthly
n=208 n=246 n=243 n=209
% respondents with the same WTP for both health states
35% 44% 43% 55% **
Skewness
BKA 0.81 ** 1.24 ** 7.64 ** 2.93 **
Paraplegia 0.07 0.72 ** 5.42 ** 2.00 **
Kurtosis
BKA 2.74 3.97 * 70.77 ** 15.19 **
Paraplegia 1.82 ** 2.61 38.68 ** 8.77 **
Spearman rank correlation coefcients for WTP and income
BKA 0.01 0.07 0.30 ** 0.33 **
Paraplegia 0.12 0.14 * 0.30 ** 0.39 **
** p<0.01, * p<0.05
income (Wilcoxin rank-sum, p<.001); three-quarters of
these cases had income less than the median. It is pos-
sible that these subjects did not have any discretionary
nancial resources with which to pay for a cure (Smith,
2005). Respondents who gave zero or missing values for
both health states were dropped from the remainder of the
analyses.
Another type of potentially questionable value came
from respondents who gave the same non-zero, non-
missing value for both health states. Table 2 shows
the distribution of these cases. Participants assigned to
a monthly format (dollar or percentage) gave the same
WTP for both health states more often than those who
were not (p=0.004). Participants assigned to a percentage
format (monthly or lump sum) gave the same WTP val-
ues for both health states less often than those who were
not (p=0.008). The combined effect resulted in only 35%
of participants who were assigned to the total percentage
format giving the same WTP for both health states while
over half (55%) of participants assigned to the monthly
dollar format did so (p<0.001).
3.3 WTP values
Table 3 shows mean and median WTP values for each of
the elicitation formats. Respondents were willing to pay
$30,276 in total or $252 per month to cure BKA when
WTP was elicited as dollars. WTP in terms of percent-
ages were 35% of nancial resources as a total amount
and 28% when elicited on a monthly basis. To cure para-
plegia, respondents were willing to pay $73,968 in total
or $325 per month; WTP, when elicited as percentages
was 53% as a total amount and 39% on a monthly basis.
3.4 Ordinal consistency of responses
On average, 88% of respondents who gave different WTP
values for the 2 health states were willing to pay more to
cure paraplegia than for BKA (Table 3). The rate of or-
dinal consistency did not vary by whether or not WTP
was elicited by month (p=0.41). However, respondents
assigned to a percentage format had a higher rate of or-
dinal consistency (91%) compared to those assigned to a
dollar format (84%) (p=0.03).
3.5 Sensitivity to differences in severity
WTP means for the two health states were signi-
cantly different, regardless of the elicitation format (p-
values<0.001). However, the differences in effect size
across the versions varied considerably. The percentage
format on a total basis had nearly a 3 times larger effect
size than the corresponding dollar format. The effect size
for the percentage format on a monthly basis was over
1.5 times larger compared to the effect size for dollars
elicited on a monthly basis. As seen in Table 3, these dif-
ferences in effect sizes translate to dramatic differences
in sample sizes needed to detect differences between the
two health states.
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 101
Table 3: WTP values by version
WTP elicited as:
% Total % Monthly $ Total $ Monthly
BKA
1
Mean 35 28 30,276 252
Median 25 20 8,500 150
(s.d.) (28) (23) (91,947) (289)
Paraplegia Mean 53 39 73,968 325
Median 50 30 10,000 200
(s.d.) (30) (27) (209,814) (324)
% respondents willing to pay more to cure paraplegia than to cure BKA
1
93% 88% 84% 84%
Cohens d effect size
2
0.70 0.57 0.24 0.35
Sample size required
3
16 25 137 64
1. Below-the-knee amputation. Include only respondents who gave different WTP values
for the two health states.
2. Effect size, used in power analyses, for comparing difference in mean WTP for BKA
and paraplegia for each of the elicitation versions.
3. Sample size that would be needed to detect the difference in mean WTP with 80% power
and 5% alpha level for each of the elicitation versions.
3.6 Normality of responses
As can be seen in Table 3, there is a wide disparity be-
tween mean and median values, especially for the dollar
amount formats, indicating highly skewed distributions.
Indeed, Table 2 shows that the skew statistics for the dol-
lar value formats were 2.0 or higher, indicating a distri-
bution that is skewed toward high positive values. The
skew statistics for 3 out of 4 of aggregate values using
percentage formats were less than 1.0. However, the only
distribution of responses that was statistically similar to
a normal distribution were WTP values elicited in terms
of the total percentage of nancial resources for curing
paraplegia (p=.7). Most response distributions exhibited
signicant kurtosis, with kurtosis statistics as high as 71
for WTP values expressed as dollars. A normally dis-
tributed set of responses would have a statistic equal to
3.0. WTPs in terms of percent of nancial resources are
much closer to this target value and in fact, 2 of the 4 sets
of responses are statistically similar to that expected for a
normal distribution (p-values>0.2).
3.7 Correlation with income
WTP expressed as absolute dollars, in monthly and to-
tal timeframes, were both signicantly correlated with
income for below-the-knee amputation and paraplegia.
These correlations were all signicantly higher than cor-
relations obtained by using the percentage formats (p-
values<.01), except that the lump sum dollar format was
only marginally higher than using the monthly percent-
age format when eliciting values for curing paraplegia
(p=.06). WTP expressed in terms of percentage of nan-
cial resources was signicantly correlated with income
only for paraplegia and only if expressed on a monthly
basis.
4 Discussion
Asking people to give their WTP as a percentage of -
nancial resources instead of asking for WTP as dollars is
a promising way to improve WTP measures that are typ-
ically plagued by undesirable properties. We also evalu-
ated timeframe and found that the advantages of the per-
centage format persisted when a per month instead of
a lump sum method was used. The percentage lump sum
format yielded the fewest respondents who gave the same
value for two different health states with clearly different
levels of severity and yielded the highest rate of respon-
dents who were ordinally consistent (WTP was higher for
curing the health state with the more severe impairment
[paraplegia] than for the less severe physical impairment
[BKA]). The two percentage formats were substantially
more sensitive to differences between health states and
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 102
thus more statistically efcient compared to WTP ex-
pressed as absolute dollars in total or on a monthly ba-
sis. This improvement in sensitivity translates to an 8-
fold reduction in the sample size required to detect com-
parable differences in other studies when comparing the
best performing format (WTP as a total percent of nan-
cial resources) to the worst performer (WTP as total dol-
lars). Both percentage formats yielded more nearly nor-
mally distributed WTP values compared to WTP in either
monthly or total dollars. The worst performer on every
criterion was WTP expressed as absolute dollars; either
monthly or total, depending on the criteria. The superior
psychometric properties assessed in this study for WTP
measured as a percent are good news considering that
though many researchers recognize the challenging dis-
tribution properties of WTP values used in CBAs (cost-
benet analyses), there has been little consensus on what
to do about it (Donaldson, 1999).
On average, participants were willing to pay 28% of
their nancial resources on a monthly basis (35% on a
total percentage basis) to cure BKA and 39% (53% on
a total percentage basis) to cure paraplegia in our study.
The percentage for curing BKA is higher than the 17%
(Thompson, Read, & Liang, 1984) and 22% (Thomp-
son, 1986) for relief of arthritis symptoms in the stud-
ies by Thompson. Schiffner and colleagues also elicited
WTP directly as a proportion of monthly income. Pre-
treatment, psoriasis patients were willing to pay 14% of
their income for a cure (Schiffner et al., 2003). It is dif-
cult to assess whether the values obtained in our study are
out of line with these previous studies because of differ-
ences in severity between the health states evaluated and
the myriad differences in elicitation methods among the
four studies.
4.1 Distributional issues
Distributional properties of WTP expressed as absolute
dollars are in line with results from other studies. Most
studies, along with this one, make note of a positively
skewed distribution of WTP expressed in absolute dol-
lars and use non-parametric approaches or mathematical
transformations prior to analyses to reduce undue inu-
ence of high values. Our skewness statistics, ranging
from 2.02.9, for monthly WTP expressed in absolute
dollars is comparable with skewness statistics from an-
other study in which WTP was elicited using an open-
ended format in an interview where participants were
asked for their WTP in terms of a weekly, fortnightly,
monthly or yearly gure. A specic timeframe was not
indicated. Skew statistics in that study ranged from 1.7
3.0 (Smith & Richardson, 2005). Even a highly skewed
measure is not necessarily invalid, but skewed measures
require transformations or use of non-parametric analy-
ses. High values may also indicate that people are giv-
ing extraordinarily high values that represent the impor-
tance of perfect health without regard for whether they
can make the tradeoffs necessary to afford the treatment.
4.2 WTP correlation with income
WTP expressed in absolute dollars clearly has a stronger
association with income than WTP expressed in terms
of percentage of nancial resources. When WTP is ex-
pressed as a percentage, the association is negligible for
both health states with both percentage formats (this is a
natural consequence if participants include their income
in considering their nancial resources). WTP expressed
as absolute dollars showed moderate associations with
income. In a recent study, WTP was less sensitive to
differences in health state, the higher the proportion of
income represented by their WTP because of personal
budget constraints (Smith & Richardson, 2005). The ex-
traordinarily high proportion of people giving the same
value for both health states when expressing WTP in a
single lump sum dollar amount may indicate that a bud-
get ceiling comes into play more readily than with the
other 3 formats; i.e., people give a WTP to cure BKA at
the maximum of what they can afford and they have no
discretionary wealth remaining to cure paraplegia even
though they may agree they would be worse off. On
the other hand, there is evidence that people are often
scale insensitive when giving WTP values these val-
ues may simply reect the respondents subjective desire
to be healthy without considering difference in severity
(Baron & Greene, 1996).
We have shown that WTP, elicited as a percentage, has
superior measurement properties. However, some may
argue that we failed to measure what needs measuring
(the amount people are willing to pay for various treat-
ment options) with this approach after all, CBAs re-
quire dollars, not percentages. We argue, however, that
WTP measured as a percentage can be readily converted
to dollar amounts in several ways, and thus provides more
exibility in addition to better measurement properties.
As with our study, Schiffner et al. (2003) and Thomp-
son et al. (1984; also Thompson, 1986) found no asso-
ciation between income and WTP when WTP was ex-
pressed as a percentage of wealth but, as with many prior
studies, we did nd that WTP elicited using absolute dol-
lars was moderately and signicantly associated with in-
come. The dissociation of WTP from income may be
cause for alarm for some economists who regard the pres-
ence of this association as one criterion by which to val-
idate the WTP values elicited (Brach et al., 2005; Don-
aldson, 1999; Donaldson et al., 1997). This may be good
news to others, however, who point out the ethical issues
that arise when WTP is associated with income out of
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 103
fear that the buying power of the rich will give them
a disproportionate voice in prioritization schemes (Olsen
& Smith, 2001). Some researchers see merit in both con-
cerns (Donaldson et al., 2002).
Percentages can be converted to dollars in two ways.
First, for those concerned about the lack of association
of income with WTP, percentages can be converted to
dollars using individual income (Klose, 1999). Measure-
ment issues aside, these dollars are the same as if elicited
directly and thus association with income will be estab-
lished while preserving the psychometric properties of
elicited percentages. In fact, backing into dollars this way
may result in WTPs that are more highly correlated with
level of income than dollars elicited directly. People may
be under-sensitive to their own ability to pay because of
the difculty of thinking about a dollar amount to pay for
the good in question and then to consider whether they
can afford that amount. The percentage format allows
people to think directly in terms of proportion of what
they can afford, thus simplifying the task.
Second, those concerned about association of WTP
with income have the option of applying the average
WTP percentage to average income of the appropriate
population (or subgroup) to obtain average WTP in dol-
lars, dissociated with income (Thompson et al., 1984),
an approach the World Bank has used to incorporate
equity considerations in CBAs of healthcare projects.
This approach incorporates distribution weighting con-
sistent with an inequality-averse society (Brent, 2003) for
healthcare. Using raw WTP expressed as a proportion of
nancial resources will result in a group with one-quarter
average income having a weight of four while those in an
income group with four times the average would have a
weight of one-quarter. However, some argue that this ap-
proach, at best, results in an index of the strength of so-
cial preferences with obscure meaning that makes WTP
elicited as a percentage of income irrelevant from the per-
spective of economic theories underlying the conduct of
CBAs (Smith & Richardson, 2005), page 82). Resolving
these differing viewpoints and challenges is beyond the
scope of this paper.
4.3 Limitations and open questions
This study has several limitations. Our scenarios did not
specify a timeframe in which payments would need to
be made nor how long the cure would last if payment
stopped. Though many studies do not spell out specic
time-periods (Smith, 2003), it is important to do so to en-
sure consistent interpretation of the elicitation and results.
We conducted this study over the Internet and had a low
initial response rate. However, once people clicked onto
the survey, 75%of themcompleted the survey and 98%of
those responses were sufciently valid to include in our
analyses. We did not intend to generalize actual WTP val-
ues obtained in this study but rather sought a diverse sam-
ple to participate in an experimental study. We were suc-
cessful in recruiting a diverse sample with respect to age,
race and ethnicity, education, and income group. In ad-
dition, these demographic characteristics were balanced
across the experimental groups. Thus, we expect that the
differences we observed in behavior with the four for-
mats in this study will extended to other similar popula-
tions. Our results were also in line with those obtained in
two pilot studies we conducted using a paper survey of a
smaller convenience sample.
WTP expressed as a percentage of monthly nancial
resources was lower than WTP expressed as a total per-
centage. Purely mathematically, the percentages should
be the same if the same sources of nances were con-
sidered in the two timeframes. However, there are many
reasons to believe this may not be the case. People may,
in fact, be drawing upon different nancial resources on
a monthly versus lump-sum basis. It would not be un-
reasonable for respondents to consider the wider range of
assets that may be available to them on a one-time lump
sum basis. They may more willing to use their borrowing
power or to dip into savings to cure their health condi-
tion with a single payment. The monthly timeframe may
more salient for many people who budget on a monthly
basis and this format may focus respondents on cash
ow where income may be the primary monthly source
of incoming cash. Relatively speaking, smaller amounts
may be available for discretionary expenditures month-
to-month, after paying for things like housing, utilities,
and food. Psychologically, shorter timeframes lead to
more concrete thinking and predictions (Trope & Liber-
man, 2003). Though WTP as a percentage of total -
nancial resources performed well based on distributional
criteria, we cannot ignore the fact that half of our respon-
dents were willing to forego half or more of their nancial
resources to cure paraplegia while, on a monthly basis,
the median amount was only 30%.
We did not actually convert WTP percentages into dol-
lars for this study. If we did so, based on our data and
assuming gross income as the denominator (the only -
nancial measure we collected in this study), values would
be signicantly higher than dollars elicited directly (for
both monthly and annual amounts). Such a comparison,
however, is fraught with issues. Dollar amounts would
likely be over-estimated because we would not be able
to take taxes into account; most people consider after-tax
income, not gross income when considering the dollars
they can afford to pay for something. However, if peo-
ple really did consider more than just their income and if
we were not constrained by a yearly timeframe, then the
converted dollars would be under-estimates. It is clear
that more study is needed to discern what respondents are
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 104
considering when giving their WTP in dollars or percent-
ages and more elaborate measures of wealth and income
are needed. The Health and Retirement Study is one ex-
ample where participants are asked for information about
many components that comprise their nancial resources
(Juster & Suzman, 1995).
The WTP values elicited in our study were for curing
relatively severe disabilities with idealized treatments.
Both of these factors led to relatively large, whole number
percentages for most participants. But the percentage for-
mat may be difcult to use when placing value on more
modest (and realistic) treatments. For example, WTP for
mammography screening was as low as $12 in one study
(Yasunaga, Ide, Imamura, & Ohe, 2007); it would be very
difcult for people to estimate such small a percentage
of annual take home income. However, there is evidence
that even when eliciting WTP in terms of dollars, low val-
ues may be less reliable than high values (Smith, 2006).
More work is needed to determine the validity of re-
sponses elicited through the Internet. Though we were
concerned about the potential for a high level of protest or
spurious responses, we did not see evidence of this. An-
other study elicited utilities for four different health con-
ditions (including BKA and paraplegia) from this same
panel of Internet users who were recruited in the same
way at the same time. The large majority of responses
were reasonable and valid. Participants gave responses
that were highly differentiated between four different
health conditions and 74% of those who gave different
utilities for BKA and paraplegia (comprising 62% of re-
spondents) gave rankings that were consistent with the
corresponding utilities (Damschroder, Zikmund-Fisher,
& Ubel, in press). Most of the questionable responses
in the present study were a result of respondents giving
the same non-zero WTP for both health states. The high
rate of equal values is troubling, but this may partly be a
function of budget constraint (Smith, 2005). The elicita-
tion format appears to inuence the rate of inconsistent
responses; evident in the lower rate of people with the
dollar formats who did not conform to our ordinal crite-
ria compared to the rate for the percentage formats. Many
researchers insist that because of the high cognitive de-
mand of WTP elicitations, in-person interviews are nec-
essary (e.g., Arrow et al., 1993). Our results are not much
different from another recent study using face-to-face in-
terviews in a large diverse sample in which 41% of par-
ticipants gave all zeros or equal non-zero WTP values for
3 treatment programs (J.A. Olsen, Donaldson, Shackley,
& EuroWill Group, 2005); a reason for some optimism
for reliably eliciting WTP values using a web-based in-
strument.
Nonetheless, the larger question of whether people
have consistent values for health conditions with which
they are not familiar has yet to be answered denitively.
Regardless of format, further work is needed to determine
the appropriate dose of information to help people dis-
cover what their true preferences are (Watson & Ryan,
2006) whether coupled with an opportunity for peo-
ple to deliberate various considerations (e.g., (Abelson et
al., 2003; Damschroder, Ubel, Zikmund-Fisher, Kim, &
Johri, 2005; Dolan, Cookson, & Ferguson, 1999), feed-
ing back an interpretation of respondents WTP so they
can afrm or change their response (Watson & Ryan,
2006), or whether researchers simply need better ways to
uncover already existing underlying preferences without
being inuenced by the method (Sugden, 2005). In ad-
dition, many psychological questions remain about what
WTP elicited using these kinds of methods actually rep-
resents. Common sources of biases have were described
earlier but in addition, regardless of format, people tend
to give the same WTP for varying levels of goods (scale
insensitivity), and WTP value for two units valued sep-
arately is often higher than WTP for 2 units valued to-
gether (lack of additivity) (Baron, 1997), WTP values are
often more reective of perceived market value or cost to
produce and not a reection of their own personal valu-
ation (Baron & Maxwell, 1996). Results from our study
help to illuminate ways to elicit consistent and valid WTP
amounts from people over the internet, but do not solve
the larger issues around WTP values, which despite chal-
lenges, continue to be used in CBAs of healthcare pro-
grams.
References
Abelson, J., Eyles, J., McLeod, C. B., Collins, P., Mc-
Mullan, C., & Forest, P. G. (2003). Does deliberation
make a difference? Results from a citizens panel study
of health goals priority setting. Health Policy, 66, 95
106.
Arrow, K., R, S., Portney, P., Leamer, E., R, R., & H,
S. (1993). Report of the NOAA panel on contingent
valuation. Federal Register, 58, 46014614.
Baron, J. (1997). Biases in the quantitative measurement
of values for public decisions. Psychological Bulletin,
122, 7288.
Baron, J., & Greene, J. (1996). Determinants of insen-
sitivity to quantity in valuation of public goods: Con-
tribution, warm glow, budget constraints, availability,
and prominence. Journal of Experimental Psychology:
Applied, 2, 107125.
Baron, J., & Maxwell, N. P. (1996). Cost of public goods
affects willingness to pay for them. Journal of Behav-
ioral Decision Making, 9, 173183.
Brach, M., Gerstner, D., Hillert, A., Schuster, A., Sos-
nowsky, N., & Stucki, G. (2005). Development and
evaluation of an interview instrument for the monetary
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 105
valuation of expected and perceived health effects us-
ing rehabilitation interventions as a model. Physikalis-
che Medizin Rehabilitationsmedizin Kurortmedizin,
15, 7682.
Brent, R. (2003). Cost-benet analysis and health care
evaluations. Cheltenham, UK: Edward Elgar.
Cohen, J. (1988). Statistical Power Analysis for the Be-
havioral Sciences (2nd ed.). Hillsdale: Lawrence Erl-
baum Associates.
Damschroder, L. J., Ubel, P. A., Zikmund-Fisher, B. J.,
Kim, S. Y., & Johri, M. (2005). A randomized trial of a
web-based deliberation exercise: improving the qual-
ity of healthcare allocation preference surveys. Paper
presented at the The 27th Annual Meeting of the Soci-
ety for Medical Decision Making.
Damschroder, L. J., Zikmund-Fisher, B. J., & Ubel, P. A.
(in press). Considering adaptation in preference elici-
tations.
Diener, A., OBrien, B., & Gafni, A. (1998). Health care
contingent valuation studies: a review and classica-
tion of the literature. Health Economics, 7, 313326.
Dolan, P., Cookson, R., & Ferguson, B. (1999). Effect
of discussion and deliberation on the publics views of
priority setting in health care: focus group study. BMJ,
318, 916919.
Donaldson, C. (1999). Valuing the benets of publicly-
provided health care: does ability to pay preclude
the use of willingness to pay? Social Science and
Medicine, 49, 551563.
Donaldson, C., Birch, S., & Gafni, A. (2002). The dis-
tribution problem in economic evaluation: income and
the valuation of costs and consequences of health care
programmes. Health Economics, 11, 5570.
Donaldson, C., Thomas, R., & Torgerson, D. J. (1997).
Validity of open-ended and payment scale approaches
to eliciting willingness to pay. Applied Economics, 29,
7984.
Haukoos, J. S., & Lewis, R. J. (2005). Advanced statis-
tics: bootstrapping condence intervals for statistics
with difcult distributions. Academic Emergency
Medicine, 12, 360365.
Juster, F., & Suzman, R. (1995). An overview of the
Health and Retirement Study. Journal of Human Re-
sources, 30, S7S56.
Kahneman, D., Ritov, I., & Schkade, D. A. (1999). Eco-
nomic preferences or attitude expressions?: An analy-
sis of dollar responses to public issues. Journal of Risk
and Uncertainty, 19, 203235.
Klose, T. (1999). The contingent valuation method in
health care. Health Policy, 47, 97123.
OBrien, B., & Gafni, A. (1996). When do the dollars
make sense? Toward a conceptual framework for con-
tingent valuation studies in health care. Medical Deci-
sion Making, 16, 288299.
Olsen, J. A., Donaldson, C., Shackley, P., & EuroWill
Group. (2005). Implicit versus explicit ranking: On in-
ferring ordinal preferences for health care programmes
based on differences in willingness-to-pay. Journal of
Health Economics, 24, 990996.
Olsen, J. A., & Smith, R. D. (2001). Theory versus prac-
tice: a review of willingness-to-pay in health and
health care. Health Econ, 10, 3952.
Payne, J. W., Bettman, J. R., & Schkade, D. A. (1999).
Measuring constructed preferences: Towards a build-
ing code. Journal of Risk and Uncertainty, 19, 243
270.
Schiffner, R., Schiffner-Rohe, J., Gerstenhauer, M., Hof-
stadter, F., Landthaler, M., & Stolz, W. (2003). Will-
ingness to pay and time trade-off: sensitive to changes
of quality of life in psoriasis patients? Br J Dermatol,
148, 11531160.
Smith, R. D. (2000). The discrete-choice willingness-to-
pay question format in health economics: Should we
adopt environmental guidelines? Med Decis Making,
20, 194206.
Smith, R. D. (2003). Construction of the contingent val-
uation market in health care: a critical assessment.
Health Econ, 12, 609628.
Smith, R. D. (2005). Sensitivity to scale in contingent
valuation: The importance of the budget constraint.
Journal of Health Economics, 24, 515529.
Smith, R. D. (2006). The relationship between reliabil-
ity and size of willingness-to-pay values: a qualitative
insight. Health Economics, 9999, n/a.
Smith, R. D., & Richardson, J. (2005). Can we estimate
the social value of a QALY? Four core issues to re-
solve. Health Policy, 74, 7784.
Sugden, R. (2005). Anomalies and Stated Preference
Techniques: A Framework for a Discussion of Coping
Strategies. Environmental and Resource Economics,
32, 112.
Thompson, M. S. (1986). Willingness to pay and accept
risks to cure chronic disease. Am J Public Health, 76,
392396.
Thompson, M. S., Read, J. L., & Liang, M. (1984). Fea-
sibility of willingness-to-pay measurement in chronic
arthritis. Med Decis Making, 4, 195215.
Trope, Y., & Liberman, N. (2003). Temporal construal.
Psychological Review, 110, 403421.
Venkatachalam, L. (2004). The contingent valuation
method: a review. Environmental Impact Assessment
Review, 24, 89124.
Watson, V., & Ryan, M. (2006). Exploring preference
anomalies in double bounded contingent valuation. J
Health Econ.
Whynes, D. K., Frew, E. J., & Wolstenholme, J. L.
(2005). Willingness-to-pay and demand curves: A
comparison of results obtained using different elicita-
Judgment and Decision Making, Vol. 2, No. 2, April 2007 Alternative approach to eliciting willingness to pay 106
tion formats. International Journal of Health Care Fi-
nance Economics, 5, 369386.
Whynes, D. K., Wolstenholme, J. L., & Frew, E. (2004).
Evidence of range bias in contingent valuation pay-
ment scales. Health Econonics, 13, 183190.
Yasunaga, H., Ide, H., Imamura, T., & Ohe, K. (2007).
Womens anxieties caused by false positives in mam-
mography screening: a contingent valuation survey.
Breast Cancer Research and Treatment, 101, 5964.
Yeung, R. Y., Smith, R. D., Ho, L. M., Johnston, J. M.,
& Leung, G. M. (2006). Empirical implications of re-
sponse acquiescence in discrete-choice contingent val-
uation. Health Economics, 15, 10771089.
Appendix: Health state descriptions
Below-the-knee amputation (BKA)
Imagine that you have a below-the-knee amputation and
have gone through the rehabilitation process. You use
a prosthetic device, an articial leg that ts well and is
fairly comfortable. Walking requires more effort, but you
get around pretty well and have only a slight limp. When
you are wearing long pants, nobody can tell that you are
using a prosthesis. Because your amputation is below the
knee, you can still participate in sports activities; you just
wont be able to run as fast or jump as high. Other than
your amputation, you are perfectly healthy.
Paraplegia
Imagine living with parapalegia. Your legs are paralyzed
from the waist down. You cannot move your legs and you
have to use a wheelchair to get around. Your bladder and
bowel functioning are both normal; however, you some-
times need help getting to the toilet. You also require help
in bathing and other daily activities. You do not have any
health problems other than paraplegia.