Evaluation of Stated Preference Surveys with Statistical Methods

Tibor SIPOS

Original Scientific Paper [email protected], Budapest University of Technology and Economics, Faculty of Transport
Submitted: 22 Mar. 2023 Engineering and Vehicle Engineering
Accepted: 11 July 2023

ABSTRACT
In this paper, the author investigated the stated preference survey in transport modelling.
The research was conducted to ensure that the best fractional orthogonal design of stated
This work is licensed preference paired comparison survey would not increase the error or uncertainty in transport-
under a Creative related decision modelling. The research was conducted based on artificial Monte Carlo
Commons Attribution 4.0
simulated respondents, and the results were assessed with standard mathematical-statistical
International License
tools. Although the assessment should have resulted in 0% errors, according to our 2,000
Publisher: sample, a minor 5% of errors occurred. The problem to be investigated in this paper is that
Faculty of Transport the best-designed survey could have some errors.
and Traffic Sciences,
University of Zagreb KEYWORDS
stated preference survey; willingness to pay; monetary value of travel time.

1. INTRODUCTION
A stated preference survey is used in economics research and social science to study consumers’ behaviour
and preferences by asking them about their choices [1]. This method could map preferences for certain goods,
services or attributes. In a stated preference survey, participants are faced with several hypothetical choices and
are asked to decide. This method differs from an actual preference survey, in which individuals are observed
making real-life choices, as in stated preference surveys, the choices are only hypothetical [2]. In this paper,
the author investigated the pairwise comparison stated preference survey. Pairwise comparison generally is
any process of comparing entities in pairs to judge which of each entity is preferred or has a greater amount
of some quantitative property or whether or not the two entities are identical [3]. The pairwise comparison
method studies preferences, attitudes, voting systems, social choice, public choice, requirements engineering
and multiagent AI systems. It is often referred to as paired comparison. Prominent psychometrician L. L.
Thurstone first introduced a scientific approach to using pairwise comparisons for measurement in 1927, which
he referred to as the law of comparative judgment. Thurstone linked this approach to the psychophysical theory
that Ernst Heinrich Weber and Gustav Fechner developed. Thurstone demonstrated that using an interval-type
scale, the method could order items along a dimension such as preference or importance [4]. Ernst Zermelo
(1928) [5] first described a model for pairwise comparisons for chess ranking in incomplete tournaments,
which serves as the basis (even though not credited for a while) for methods such as the Elo rating system and
is equivalent to the Bradley-Terry model that was proposed in 1952 [6].
In this paper, the author investigated the fractional orthogonal design of a stated preference paired comparison
survey to estimate the monetary value of travel time savings [7]. Orthogonality among the attributes allows
estimating the main effects of one variable on choice independently of the effects that the other variables may
have. For instance, the passenger car versus bus mode choice travel time difference and travel cost difference
could be investigated [8]. If the choices always have the same level for the travel time difference and travel
cost difference, it means total collinearity between the two vectors. Estimating the main effect of each of
these variables on choice would not be possible. While the fractional factorial approach can significantly
reduce the questions needed for a stated preference exercise, it typically ignores some or all interaction effects.
Stated preference surveys use willingness-to-pay approaches for particular goods or services to estimate their
perceived monetary value [9] in order to analyse the effects of different attributes or factors on consumer
behaviour. They are also commonly used to evaluate the potential impact of new products or services or to
Promet ‒ Traffic&Transportation. 2023;35(5):655-661. https://doi.org/10.7307/ptt.v35i5.259
Promet ‒ Traffic&Transportation. 2023;35(5):655-661. Transport Engineering

understand consumer preferences in the context of public policy decisions, such as assessing environmental
impacts.
The monetary value of travel time saving refers to the amount of money individuals are willing to pay
to reduce the time they spend travelling. The economic value estimates that individuals place on the time
they save due to a more efficient or quicker transportation option. The monetary value of travel time saving
can be estimated through various methods, including stated preference surveys, revealed preference studies
or models that estimate the trade-off between time and money. In stated preference surveys, individuals are
asked to state their preferences for travel scenarios, including travel times and costs. In revealed preference
studies, individuals’ travel behaviour is observed and analysed to determine their value on travel time savings.
The monetary value of travel time can vary depending on several factors: income, the purpose of the trip, the
availability of alternative modes of transportation and the time of day. It is an essential consideration in the
planning and designing of transportation systems and infrastructure, as it can help policymakers determine the
potential benefits and costs of different transportation options and make informed decisions about investment
in transportation infrastructure.
The primary aim and research question arises: how can one assess if the stated preference survey is good and
the results are reliable? The aim of this study is to develop a comprehensive analysis to assess the effectiveness
and reliability of stated preference surveys as a research tool, aiming to enhance the quality and validity of
survey results.

2. METHODOLOGY
In economics, utility measures a person’s satisfaction with a good or service or benefit from consuming a
good or service. It is a subjective concept, as different people may have different preferences and values and
therefore derive different levels of utility from the same good or service. The utility can be measured in different
ways, such as by asking people to rate their satisfaction with a particular product or service or observing their
behaviour when making choices in a marketplace [10]. One common way to measure utility is to use a utility
function, which assigns a numerical value to each possible combination of goods and services that a person
can consume based on their preferences and the prices of the goods. In general, people will try to maximise
their total utility by choosing a combination of goods and services that gives them the most satisfaction with
the amount of money they spend. This principle is the basis for many economic theories and models, including
consumer theory and utility maximisation. In this article, the following linear utility function has been used
based on the random utility theory (Equation 1):
U = a ⋅P + b ⋅TT + g ⋅CT + d ⋅ TR + e (1)
where:
a – utility weight parameter of price, sensitivity parameter [11]
P – price of travel [12]
b – utility weight parameter of travel time, sensitivity parameter [13]
TT – time of travel [14]
g – utility weight parameter of crowding, sensitivity parameter [15]
CT – crowdedness of travel, sensitivity parameter [16]
d – utility weight parameter of the number of transfers [17]
TR – number of transfers of travel, sensitivity parameter [18]
e – error term of the utility function.
The author used Monte Carlo simulation to generate the responses based on the parameter model of
fractional orthogonal stated preference survey based on the given utility functions [19]. The author has defined
a, b, g and d as probability variables with their given probability density, the Monte Carlo process assigned
randomly selected values from the given probability density functions to calculate unique personal utility [20].
After that, based on the choice of stated preference surveys, the a’, b’, g’ and d’ parameters were determined
with logit regression modelling [21]. The article aims to investigate if the probability density functions of a’,
b’, g’ and d’ could fit the original probability density function of a, b, g and d. The logit expression is well-
known in decision theory (Equation 2):
656
Promet ‒ Traffic&Transportation. 2023;35(5):655-661. Transport Engineering

1
p ( x) = (2)
1 + e− x
where p(x) is the probability of choice based on the utility function x. In order to assess the questionnaire in
this paper, the author artificially generated 2,000 respondents to evaluate the validity and reliability. The 2,000
respondents had 2,000 different utility functions, where parameters were given as a probability density function
to ensure the randomised effect. Therefore, the main question is if the assessment of survey results gives back
the theoretical probability density functions of utility parameters. To understand the problem analytically, the
following Equation 3 needs to be solved:
∞ 0
1
∫−∞ 1 + e− x dx − −∞∫ 1dx =
0 (3)

The first part is the theoretical logit model of choice. Meanwhile, the second part is the reverse-engineered
decision function. It can be easily seen that both figures cover the same area geometrically (Figure 1).
p

B
-4 -3 -2 -1 1 2 3 4 A-B

Figure 1 – Visualisation of logit function and stated choice

Therefore, the ideally constructed fractional orthogonal preference survey should give back the distribution
parameters of the utility function with no error. As generally accepted, the goodness of the survey was measured
by the coefficient of determination (Equation 4):
n

∑ ( y − yι )
2
i
R2 = 1 − i =1
n
(4)
∑ ( y − yι )
2
i
i =1

where:
yi – AI-generated parameter
ŷι – revealed parameter from regression analysis
yι– arithmetic average of AI-generated parameters.
Several other factors can be used to assess the goodness of a stated preference survey.
Validity. The validity of a stated preference survey refers to its ability to accurately measure, in this case,
the estimated parameters of the utility function. Validity includes the accuracy of the hypothetical scenarios
presented to participants and the questions used to elicit their preferences.
Reliability. The reliability of a stated preference survey refers to its ability to produce consistent results
when repeated with the same sample of participants. This is important for ensuring that random error or
measurement bias does not affect the results.
Representativeness. The representativeness of a stated preference survey refers to its ability to reflect the preferences
of the population of interest accurately. Representativeness includes ensuring that the sample of participants is
representative of the population and that the hypothetical scenarios presented reflect real-life situations.
Responsiveness. The responsiveness of a stated preference survey refers to its ability to detect changes in
preferences over time or in response to different circumstances. Responsiveness is essential for ensuring that
the results remain relevant and up-to-date.
Transparency. The transparency of a stated preference survey refers to the degree to which its methods and
results are clearly and openly presented so that they can be easily understood and evaluated.
Feasibility. The feasibility of a stated preference survey refers to its practicality and ability to be implemented
effectively in terms of cost, time and resources.
657
Promet ‒ Traffic&Transportation. 2023;35(5):655-661. Transport Engineering

3. RESULTS
A utility function is a mathematical representation of an individual’s preferences over goods, services or
outcomes. It quantifies an individual’s satisfaction or happiness from consuming a particular combination
of goods or services. In economics, the utility function is a mathematical equation that maps inputs (such as
the quantities of different goods or services consumed) to a single value representing the individual’s total
satisfaction or utility. The utility function allows economists to study and analyse consumer behaviour and
predict how consumers will respond to price changes or the availability of goods and services. The utility
function is usually assumed to satisfy specific properties, such as being non-negative, monotonically increasing
and satisfying the law of diminishing marginal utility. These properties reflect the idea that as an individual
consumes more of a good, their marginal satisfaction (the additional satisfaction derived from consuming one
more unit) will decrease, reflecting the concept of diminishing returns. In practice, the utility function is often
estimated using data from surveys or experiments, such as stated preference surveys, which ask individuals to
0.115
0.11
Value of alpha

0.105
0.1
Alpha_AI
0.095
Alpha_Reg
0.09
0.085
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Number of respondents
a) Visualisation of alpha parameter

1000
800
Value of beta

600
400 Beta_AI
200 Beta_Reg

0
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Number of respondents
b) Visualisation of beta parameter

160
140
Value of gamma

120
100
80
60 Gamma_AI
40 Gamma_Reg
20
0
0 500 1000 1500 2000
Number of respondents
c) Visualisation of gamma parameter

0.4
0.2
Value of delta

0
-0.2 Delta_AI
-0.4 Delta_Reg
-0.6
-0.8
Number of respondents
d) Visualisation of delta parameter
Figure 2 – Visualisation of AI-generated and the revealed parameter from regression analysis

658
Promet ‒ Traffic&Transportation. 2023;35(5):655-661. Transport Engineering

state their preferences for different combinations of goods and services. The resulting utility function can then
predict how individuals will allocate their resources given different prices or availability of goods and services.
The author has defined a, b, g and d as probability variables with their given probability density, the Monte
Carlo process calculated unique personal utility functions. After that, based on the choice of stated preference
surveys the utility parameters were determined with logit regression modelling. Here one can see the four
utility parameters for the 2,000 respondents as created by AI and reverse-engineered by regression (Figure 2).

0.13
Regression estimation

0.12
0.11
0.1 • Alpha
.......... y=x
0.09
0.08
0.08 0.085 0.09 0.095 0.1 0.105 0.11 0.115 0.12 0.125

Artificial intelligence
a) Goodness of fit alpha

1200
Regression estimation

1000
800
600
• Beta
400 .......... y=x
200
0
0 200 400 600 800 1000 1200
Artificial intelligence
a) Goodness of fit beta

160
Regression estimation

140
120
100
80
60
• Gamma
.......... y=x
40
20
0
0 200 400 600 800 1000 1200 1400 1600
Artificial intelligence
a) Goodness of fit gamma

0.7
Regression estimation

0.5
0.3
0.1
• Delta
-0.5 -0.3 -0.1 -0.1 0.1 0.3 0.5 0.7
.......... y=x
-0.3
-0.5
Artificial intelligence
a) Goodness of fit delta

Figure 3 – Visualisation of AI-generated and the revealed parameter from regression analysis

659
Promet ‒ Traffic&Transportation. 2023;35(5):655-661. Transport Engineering

By analysing the values of these utility parameters, insights can be gained into respondents’ preferences and
sensitivities regarding price, travel time, crowding and the number of transfers. The graph serves as a visual
representation of these parameters, providing valuable information for transportation planning and decision-
making processes.

4. DISCUSSION
This part describes the statistical analysis of AI-generated and the revealed parameter from regression
analysis. As shown in Figure 3, the revealed parameters fit well to an identical line.
The goodness of the survey was measured by the coefficient of determination (Table 1).
Table 1 – Coefficients of determination of parameters

Name of parameter Alpha Beta Gamma Delta

R 2
0.99 0.91 0.93 0.92

As stated, the reconstruction of the parameters derived from the fractional orthogonal stated preference
survey was successful. In addition, the revealed parameters are very close to the artificially generated (Table 2).
Table 2 – Descriptive statistic of comparison

Name of parameter Alpha Beta Gamma Delta

Difference in average 0.004% -0.447% -0.152% -0.122%
The difference in standard deviation 0.002% -4.175% -3.364% -4.395%

In this paper, the author investigated the pairwise fractional orthogonal stated preference survey to determine
the monetary value of travel time. Based on the literature, the monetary value of travel time can be estimated
based on the sensitivity parameter price and the sensitivity parameter of travel time (Equation 5).
α
WTP = (5)
β
5. CONCLUSION
When conducting research, there is always a considerable risk if fractional orthogonal stated preference
surveys can appropriately measure the investigated process. Are the survey results valid and can they be
trusted? What is the limitation of such surveys?
α
β
∆WTP = (6)
α′
β′
The survey proved to be good and the aggregated monetary value of travel time only changed up to 0.5%
(Equation 6). The proposed process can easily be transferred to any fractional orthogonal stated preference
survey if the theoretical probability density function of utility sensitivity parameters is well-known.

ACKNOWLEDGEMENT
The research was supported by OTKA-K20-134760-Heterogeneity in user preferences and its impact on
transport project appraisal supervised by Adam Torok. This work was supported by Daniel Tordai, PhD student
at Budapest University of Technology and Economics, Department of Transport Technology and Economics.

REFERENCES
[1] Babic D, et al. Choice factors of distribution channels. J. Transp. Logist. 2010;1:5-13. https://journals.vstecb.cz/
wp-content/uploads/fullissue/35.pdf [Accessed 20th Jan. 2018].
[2] Babojelić K, Novacko L. Modelling of driver and pedestrian behaviour–a historical review. Promet –
Traffic&Transportation. 2020;32(5):727-745. DOI: https://doi.org/gp7q6h.
[3] Vij A, Krueger R. Random taste heterogeneity in discrete choice models: Flexible nonparametric finite mixture
distributions. Transportation Research Part B: Methodological. 2017;(106):76-101. DOI:
https://doi.org/gcqbd3.
660
Promet ‒ Traffic&Transportation. 2023;35(5):655-661. Transport Engineering

[4] Thurstone LL. A law of comparative judgement. Psychological Review. 1927;(34):278-286. DOI:
https://doi.org/b9pn6t.
[5] Zermelo E. The calculation of tournament results as a maximum problem of probability theory [Die Berechnung
der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung]. Mathematical Journal
[Mathematische Zeitschrift]. 1928;(29):436-460 DOI: https://doi.org/bvrnmg.
[6] Bradley RA, Terry ME. Rank analysis of incomplete block designs, I. The method of paired comparisons.
Biometrika. 1952;(39):324-345. DOI: https://doi.org/c5bcq8.
[7] Train K. Mixed logit with a flexible mixing distribution. J. Choice Model. 2016;(19):40-53. DOI:
https://doi.org/grgzg6.
[8] Novačko L, et al. Selection of LRT system track gauge using multi-criteria decision-making (City of Zagreb). WIT
Transactions on the Built Environment. 2008;101(7):167-173. DOI: https://doi.org/dhbgs8.
[9] Novačko L, et al. Simulation-based public transport priority tailored to passenger conflict flows: A case study of
the city of Zagreb. Applied Sciences. 2021;11(11):4820. DOI: https://doi.org/gp5j7b.
[10] Keane M, Wasi N. Comparing alternative models of heterogeneity in consumer choice behavior. Journal of
Applied Econometrics. 2013;28(6):1018-1045. DOI: https://doi.org/ghnztj.
[11] Zhang Q, et al. Time differential pricing model of urban rail transit considering passenger exchange coefficient.
Promet – Traffic&Transportation. 2022;34(4):609-618. DOI: https://doi.org/ksb9.
[12] Vasudevan N, et al. Determining mode shift elasticity based on household income and travel cost. Research in
Transportation Economics. 2021;(85):100771. DOI: https://doi.org/gmt56k.
[13] Hensher DA. The sensitivity of the valuation of travel time savings to the specification of unobserved effects.
Transportation Research Part E: Logistics and Transportation Review. 2001;37(2-3):129-142. DOI:
https://doi.org/cw38hj.
[14] Jiang R, et al. Predicting bus travel time with hybrid incomplete data – A deep learning approach. Promet –
Traffic&Transportation. 2022;34(5):673-685. DOI: https://doi.org/kscb.
[15] Sadrani M, et al. Optimisation of service frequency and vehicle size for automated bus systems with
crowding externalities and travel time stochasticity. Transportation Research Part C: Emerging Technologies.
2022;143:103793. DOI: https://doi.org/jnm3.
[16] Bansal P, et al. A dynamic choice model with heterogeneous decision rules: Application in estimating the user cost
of rail crowding. arXiv preprint arXiv. 2020. DOI: https://doi.org/kscc.
[17] Massobrio R, et al. Learning to optimise timetables for efficient transfers in public transportation systems. Applied
Soft Computing. 2022;119:108616. DOI: https://doi.org/kscd.
[18] Manasra H, Toledo T. Optimisation-based operations control for public transportation service with transfers.
Transportation Research Part C: Emerging Technologies. 2019;105:456-467. DOI: https://doi.org/kscf.
[19] Aboutaleb Y, et al. Discrete choice analysis with machine learning capabilities. arXiv preprint arXiv. 2021. DOI:
https://doi.org/kscg.
[20] Bansal P, et al. Flexible estimates of heterogeneity in crowding valuation in the New York City subway. Journal of
choice modelling. 2019;31:124-140. DOI: https://doi.org/gh55rs.
[21] Bansal P, et al. Comparison of parametric and semiparametric representations of unobserved preference
heterogeneity in logit models. Journal of Choice Modelling. 2018;27:97-113. DOI: https://doi.org/gdmm2w.

Sipos Tibor
Feltárt preferencia alapú kérdőív statisztikai elemzése

Absztrakt
A szerző a közlekedésmodellezésben alkalmazott feltárt preferencia alapó felmérés
elemzési lehetőséeit vizsgálta. A kutató célja célja biztosítani, hogy az ortogonális tervezésű
kérdőívekben ne növelje a hiba vagy bizonytalanság mértékét a közlekedéssel kapcsolatos
döntésmodellezésben. A kutatást mesterséges Monte Carlo szimuláción alapuló válaszadókon
végezték, és az eredményeket standard matematikai-statisztikai eszközökkel értékelték ki.
Bár az értékelésnek 0% hibát kellett volna eredményeznie, a 2000 minta alapján 3% hiba
jelentkezett. A tanulmányban azonosított probléma, hogy a legjobban tervezett felmérés is
tartalmazhat hibákat.

Kulcsszavak
feltárt preerncia; fizetési hajlandóság; utazási idő veszteségértéke.

661