Limitations on the Use

of Mathematical Models
in Transportation Policy Analysis

ABSTRACT:
Government agencies
are using many kinds
of mathematical models
to forecast the
effects of proposed
government policies.
Some models are useful;
others are not; all
have limitations.
While modeling can
contribute to effective
policyma king, it can
con tribute to poor
decision-ma king if
policymakers cannot
assess the quality of a
given application.
This paper describes
models designed for
use in policy analyses
relating to the
automotive transportation
system, discusses
limitations of these
models, and poses
questions policymakers
should ask about a
model to be sure its
use is appropriate.
 Introduction

Mathematical m o d e l i n g of real-world use in analyzing the medium and long-range ef-

systems has increased significantly in the past fects of federal policy decisions. A few of them
two decades. Computerized simulations of have been applied in federal deliberations con-
physical and socioeconomic systems have cerning policies relating to energy conserva-
proliferated as federal agencies have funded tion, environmental pollution, automotive
the development and use of such models. A safety, and other complex issues. The use of
National Science Foundation study established mathematical models in policy analyses re-
that between 1966 and 1973 federal agencies quires that policymakers obtain sufficient infor-
other than the Department of Defense sup- mation on the models (e-g., their structure,
ported or used more than 650 models limitations, relative reliability of output) to make
developed at a cost estimated at $100 million informed judgments concerning the value of
(Fromm, Hamilton, and Hamilton 1974). Many the forecasts the models produce. The ap-
more models have been developed since 1973. propriate use of models and their output can
contribute to effective policymaking, but mis-
use of models or misinterpretation of their out-
The appropriate use of madets and put can mislead decision-making.
their output can contribute to
efferctive policymaking, but misuse
of rn~ddsor misEnterpretatSan sf What is a Model?
their output can mislead decision-
making. A model is a representation of reality. Neces-
sarily it is a simplification or abstraction. A
model may be a physical representation, for ex-
The success of models that simulate physical ample, a globe. A mathematical model differs
systems has often been dramatic. A widely from the more tangible physical model, in that
known example is the modeling that the "reality" is represented by an equation or series
National Aeronautics and Space Administra- of equations. There are many kinds of models.
tion conducted in its lunar exploration pro- This paper is concerned with mathematical
gram. Computer-simulated "landings" and models, in particular, econometric models.
"retrievals" were conducted hundreds of times Econometric models have their basis in eco-
before the first manned landing was attempted. nomic theory, are derived using statistical tech-
Successful simulation of physical systems niques, and are used in studying relationships
has encouraged development of mathematical among economic variables.
models of social/economic/political "systems." Two important elements of equations are
For example, mathematical models are used to variables and parameters. Variables represent
forecast such economic indicators as the gross the elements of the system being modeled (e.g.,
national product, capital investment rates, the number of automobiles in the United
employment rates, federal tax revenues, and States). In a mathematical model the values of
other measures of the national economy. These some variables are specified outside the model.
socioeconomic models are designed to project These variables are called exogenous
or forecast the future behavior of real-world variables. The values of other variables are
systems under scrutiny. While physical prin- calculated within a model. These variables are
ciples are well understood and stable enough to called endogenous. Knowing which variables
be predictable, social, economic, and political are exogenous and which are endogenous can
behavior is not well understood, not stable, and be important in understanding the results of a
not very predictable (with some exceptions) ex- model. This is discussed later.
cept within broad limits. Parameters of an equation are factors that
Some relatively new forms of mathematical qualify the variables. For example, one might
models have been developed in recent years for calculate the number of large-size cars sold in a
year as a function of the annual income of car
buyers and their tendency to buy large cars. of and assumptions about the real world. These
People with higher incomes might purchase observations and assumptions support the
more large-size cars than people with lower in- modeler's selection of variables, parameters,
comes. Thus, a simple form of an equation to functions, and the basic logic of a model. Some
calculate large-size car sales might equate models represent systems whose behavior is
sales with some number "a" times the number well understood. An example is an electrical cir-
of people with high incomes, plus some number cuit. Observations and assumptions con-
"b" times the number of people with low in- cerning the behavior of such systems are ex-
comes. In this example, the numbers chosen for plicit and objective.
"a" and "b" are the parameters of the equation.
The interpretation of the parameters depends
on the specification of the equation structure,
that is, on the mathematical form of the equa- An inherent limitation of models is
tion. The specification of the equations and the that judgments are necessary in
derivation of the values of the parameters are building them.
i m p o r t a n t tasks i n creating a model.
Parameters remain constant for a particular
analysis, while the values assigned to variables
change. As information about real-world systems
The example above involves only one equa- becomes less precise or harder to measure,
tion. Usually, a modeler needs to address more more assumptions must be made. Modeling
than one question at the same time. For exam- becomes a less precise endeavor as it moves
ple, the modeler may wish to predict both the away from physical systems and toward social
demand for automobiles and fuel consump- systems. Modeling an electrical circuit is a
tion. Since these variables are related to each straightforward task, compared to modeling
other, a more complex model with more than human decision-making. Also, the nature of the
one equation may be required. A modeler might information about physical and human systems
create a model in which automobile demand is is different. Good historical information about a
a function of several variables (e.g., income) physical system is quite valuable in modeling
and in which fuel consumption is a function of future performance, because the system usual-
auto demand, plus several other variables (e.g., ly does not change. Good information about a
the fuel economy of the automobiles sold). The social system is equally desired but may be of
specification of the equations and the relation- less value in forming assumptions, because
ships among the various equations that social systems often change and in ways that
describe how the variables are linked in the real were not part of the past. Thus, to understand a
world represent the overall logic of the model. model's limitations it is important to under-
An analyst needs to understand the logic of a stand the assumptions that were used to create
model in order to use it intelligently. it.
When constructing models, model builders Mathematical models have been used pre-
usually experiment with a wide range of alter- dominantly in two ways in studies of the auto-
native forms to find the closest fit of the equa- mobile transportation system-forecasting and
tions to the sample data. Undue emphasis on policy analysis.
close fit, however, sometimes leads to mis- When a model is used primarily for fore-
specification of the model structure, for exam- casting, the user exercises the model to pro-
ple, by use of spurious variables in an equation duce a forecast based on the general assump-
to improve its fit. When used for forecasting or tion that past relationships among variables will
policy analysis, such a misspecified model is continue. Often this is done to identify future
poorer than a proper specification that fits less problems that may occur if past relationships
closely to sample data. continue.
A model is obviously based on observations Policy analysis applications also produce
forecasts, but the concern is with the different
futures associated with different policy assump-
tions. Policy changes are imposed on the model
and forecasts made to assess the effects of the An important inh
changes. a model is creat
For example, a forecasting use of an out.
automobile demand model would involve
specification of the exogenous variables and in-
put using the best information available to the
user. The primary interest is to estimate future effectiveness in reproducing values over some
automobile demand as it is likely to occur under historical period where the output values are
future conditions. A policy analysis application known. After this capability of the model is
might examine the expected effects on future tested, it is placed into use.
automobile demand of increases in gasoline The steps clearly are complicated but, as
prices caused by an increased federal tax. In stated above, each step may seem relatively
both cases, the model output-a forecast-is of straightforward. In reality this is not the case.
interest. Each step requires judgment. It is difficult to
Models that produce forecasts are usually represent real-world systems in terms of
designed to produce either short-term or long- mathematical relationships. Data are often
term forecasts. The assumptions, structure, and unavailable or inaccurate. Combining the sub-
associated factors required for the one purpose system models to create the model is seldom
often make the model less suitable for the simple.
other. Models designed for long-term fore- Assumptions and estimates must be made at
casting often do not produce good short-term almost every step of the process. In even the
forecasts and vice versa. best models of social systems, the biases of the
model builder are incorporated directly into the
model as the necessary judgments are made.
Building a Model The fact that judgments are necessary is an in-
herent limitation. Thus, one who uses a model
Creating a model requires completing a must understand the judgments and assump-
series of steps. Obviously, these differ some- tions associated with the model and how they
what, depending on the type of model. There influence the model performance. This requires
are, however, common elements that need to that those who build models make their judg-
be understood in order to understand how the ments explicit and document them fully. Ade-
limitations inherent in a model come into being. quate model documentation is crucial because
The steps start as the modeler specifies the it is often the only link between the model and
general elements and relationships of the real- the model user.
world system to be modeled. Data require-
ments and the availability of data are then as-
sessed. Next, submodels of the subsystems of
the real-world system are formulated. These
submodels are combined to create the larger
model. As these submodels are created and
c o m b i n e d , data are gathered and t h e
parameters of the model are estimated.
Complex models using many equations are
usually prepared in a form suitable for use on a
computer, This is a complex process also. Once
the model is built, it must be tested to establish
its validity. This may be done by testing its
I
I0 Limitations of Models 0 0E lI I I
Models have a definite role in policy analysis. sales (e.g., to governments, companies, and
They are important and powerful analytical leasing firms). Fleet purchases have been esti-
tools that can add clarity and insight to many mated to account for about twelve to twenty
analyses. But they must be used carefully. Their percent of new car sales. In recent years this
appropriate use is most likely to flow from percentage has increased.
understanding their limitations and ensuring Fleet purchases differ from individual pur-
that such limitations are considered in any chases. For example, in 1976, fleet buyers
analysis. bought fewer large-size cars (six percent of fleet
We have previously spoken of "limitations" in purchases as compared to twenty-nine per-
a general sense. At this point, we present a cent of all sales). Fleet buyers have also reacted
series of examples of limitations that are as- more quickly to external influences, like the oil
sociated with types of models commonly used embargo, than have individual purchasers. In
in motor vehicle transportation system policy 1972 only about two percent of fleet purchases
analyses. For the most part, we address prob- were small cars. In 1975, following the oil em-
lems related to use of econometric models, bargo, the percentage jumped to thirty-two per-
because they have been the most frequently cent.
used. Many of the limitations noted, however, Other differences also exist between fleet use
apply also to other types of models. and personal use. Fleet cars are driven more
For each limitation identified, the problem is and sold more quickly.
stated, an illustration presented, and the
significance discussed. As these selected Significance
limitations are only examples, our list is clearly Present automobile models in common use
not inclusive. do not attempt to explain a significant part of
the system they purport to model. Automobile
demand predictions for all sales are based on
Limitation 1. Models are Incomplete factors traditionally associated with approx-
- --
imately eighty percent of the sales.
Models are abstractions of reality. Real-world Automobile demand is not the only output af-
systems are complex and composed of many fected. Attempts to compute fuel consumption
interrelated components. A "good" model must may also be biased by the different use patterns
attempt to capture all the critical elements of the of fleet-owned vehicles.
real-world system. This is something that is vir- Because existing models ignore an important
tually impossible to do in modeling social part of reality, the significance of their output
systems. Thus, an important inherent limitation can be understood only if the influence of fleet
of a model is created by what is left out. purchases and use are understood.
The limitation may not be significant if the
omitted elements are not of great importance.
Difficulty will arise, however, if key aspects of Limitation 2. Models Assume the Future
the real-world system are left out or inade- will be Like the Past
quately treated in a model.
Models are created to represent a system
lllus tra tion based on historical information. Yet, models are
Automobile demand models typically predict used to forecast what will happen in the future.
total new car sales or registrations. These pre- If there are significant changes in the real-world
dictions are calculated as a function of variables system, the quality of the model suffers.
associated with personal automobile purchases This is a significant problem for modelers of
(e.g., family income, family size, etc.). social systems. Rapid technological, eco-
Recent data (Shonka, Loebl, and Patterson nomic, social, and political change has been a
1977) show that a significant percent of new car characteristic of modern times. If the changes
sales are not personal purchases but are fleet
that occur are not reflected in the relationships based heavily on estimates of how much it
that exist in the model, significant limitations would cost the companies to redesign cars to
result. achieve better gas mileage. The cost estimates,
in turn, depend on projecting future fuel
lllustra tion economy from historical data to determine how
The focus of automobile demand models now much improvement would be required over
in use for federal policy analysis is on the present fuel economy to meet the 1985 stan-
domestic demand for vehicles. Excluding Cana- dard.
dian sales, past exports have accounted for less
than two percent of annual sales. Thus, the lack
of an export sector in auto demand models has
not been considered a significant limitation. Uncertainty surrounding the values
The shift to small cars for the domestic of the exogenous input variables
market, the devaluation of the dollar, and the in- compounds the difficulty of deter-
creased demand for vehicles in the world mining the accuracy of model out-
market has led to speculation that U.S. exports put.
will increase in the future. U.S. small cars ap-
pear likely to become competitive in the "world
car" market.
The data used to measure fuel economy were
Significance
based on test information from the Environ-
The existing models cannot predict the ef-
mental Protection Agency (EPA). The model
fects of significant increases in exports,
authors knew that the EPA data were inaccurate
because these relationships are not included in
and historically had overstated actual fuel
the models. Nor will it be easy to include such
economy. The authors adjusted the EPA figures
relationships because of the paucity of relevant
on the basis of the best available information.
data. Thus, auto demand predictions contain an
The adjustment reduced the difference
unknown error associated with the future export
between the standard specified and the pro-
market for U S , automobiles.
jected fuel economy. Using the adjusted data,
the model forecasts supported the DOE conclu-
sion.
Limitation 3. Data Problems Careful examination of later information in-
dicated that the adjustments were too small.
Mathematical models are dependent on data, They still resulted in an overestimate of the dif-
both data used as input to the models, as dis- ference between actual on-the-road fuel
cussed in Limitation 5, and also data used in economy and the fuel economy standard. The
building the models. overestimate of this difference resulted in an
The data used in building a socioeconomic overestimate of the costs to the automakers re-
model may be incomplete and are usually es- quired to achieve the 1985 standards.
timates. For example, the U.S. Census data, The data were again revised to reduce this
even for Census years, are estimates, and the overestimate. When the revised model was run
data for the intervening years are interpolations using these data, the results showed that a fuel
of the estimates. economy level very close to the 27.5 miles per
gallon standard could be achieved in 1985 at a
lllustra tion cost no more than the penalties that would
In 1978, the U.S. Department of Energy pre- otherwise be imposed. These results produced
sented output from the Faucett Automobile a clarification of the original Department of
Sector Forecasting Model (Difiglio and Kulash Energy statements.
1976) in a report to Congress (Energy Informa-
tion Administration 1977). On the basis of the Significance
model output and other information, the DOE The consequence of data problems in this
concluded that U.S. automobile companies case is clear. An inaccurate report was sub-
would choose not to meet the 1985 fuel mitted to Congress. Fortunately, Department of
economy standard of 27.5 miles per gallon and, Energy analysts conducted more detailed
instead, would pay the penalties provided by examinations and revised the original position.
law (Kelderman 1978). The forecasts were
Despite this, the fuel economy data in the model received information labeled as an output when
are still "adjusted data" and the accuracy of the in fact it was an input. The information is not
output is dependent on the accuracy of these available to assess the significance of this par-
data. ticular event. Common sense suggests that
policymakers would be likely to place different
weight on information known to be a simple
Limitation 4. The Operational Status of a estimate than on information believed to have
Model may be Unclear been derived from a more formal forecasting
process.
One of the major problems facing potential
users is determining the model's status. Models
take months and sometimes years to build, and Limitation 5. Input Data mav be Uncertain
many model builders frequently revise their
models. At any one time, several versions of a Problems with data used in building models
model may exist. Frequently, full documenta- were discussed above. Another type of data
tion does not exist because of the dynamic problem stems from exogenous input to the
nature of model development. Thus, a user is model. Uncertainty surrounding the values of
faced with the need to determine the exact the exogenous input variables compounds the
characteristics of the version of the model being difficulty of determining the accuracy of model
used before the results can be understood. output. Future-year values of these variables
While this seems obvious, it is not always simple are forecasts, often from other models, and the
to do, and thus it is not always done. accuracy of these values is uncertain.
The types of exogenous inputs that are
lllus tra tion typical in auto demand/gasoline consumption
A part of the study conducted by the Inter- models are trends in population,, unemploy-
national Trade Commission (ITC) for the Senate ment, and gross national product. In the case of
Finance Committee to study the proposed "gas current automobile models, exogenous fore-
guzzler" tax is a classic example of this prob- casts of economic trends are usually taken from
lem (US. International Trade Commission the results of running an econometric model of
1977). One important question addressed was the national economy. For population, trend
the effect of the proposed energy policies on estimates are available from the U.S. Bureau of
the potential increases in sales of foreign auto- the Census. Sometimes there is little informa-
mobiles in the United States. In essence, fore- tion available on which to base an important
casts of the relative market shares of domestic trend, an expert opinion may be the only guide.
and foreign manufacturers were sought under The most sophisticated method of setting ex-
different policy alternatives. ogenous trends cannot remove the uncertainty
The ITC analysts used the Wharton EFA of these forecasts. This uncertainty is greater
Automobile Demand Model (Schink and Loxley the further into the future a forecast is carried.
1977) in their study, but were unaware that, in The values the model builder or the user im-
the version they were using, the splits between poses on the model can greatly affect the output
foreign and domestic shares by type of car were of the model. Knowing what exogenous data
set as exogenous variables. The splits had been and assumptions have been used, determining
preset by the model authors. This point was whether they are reasonable, and finding out
mentioned most obscurely in the model how much the forecasts from the model would
documentation. A later version of the model in- change if different data or assumptions were
cludes these foreign/domestic splits as en- used is crucial in using any model.
dogenous variables. Exogenous input may take the form of as-
sumptions specified by the user. These may
Significance represent quantifications of key aspects of pro-
The model output provided estimates of the posed alternative policies. An example of this
foreign and domestic market shares. These type of exogenous data in an auto de-
estimates were not calculated by the model. mandlgasoline consumption model is gasoline
They were specified a priori-a fact that was un- price. Future-year values of this type of variable
known to the analysts who treated the results as may either be assumed by the model user or
forecasts. Thus, the Senate Committee may be obtained from some existing source (of
unknown accuracy).
 If the model is sensitive to the exogenous
variables, it will produce significantly different Limitation 6. The Usefulness of a Model
results for different values of the input data.
may be Limited by its Original Purpose
Most models are developed for specific pur-
poses and reflect the performance or behavior
Correct use of a model requires of particular systems. Correct use of a model
knowing and understanding the requires knowing and understanding the pur-
purposes of the model. poses of the model. Failure to do this can lead
to unsuccessful or improper application of a
model or misinterpretation of its results.
Models are adaptable to uses other than the
//lustration ones they were originally designed for. This
This problem is illustrated by an example adaptability is one of the great attractions of
presented in a Jack Faucett report to the models. The adaptation of a model usually in-
Federal Energy Administration. The report volves adding to it or restructuring parts of it.
compared output of the Faucett Automobile This is a task that requires considerable care
Sector Forecasting Model in which two dif- and technical expertise. The extra cost of
ferent assumptions concerning the price of adapting a model is usually not large com-
gasoline were used. In one case gasoline price pared to the original cost of building it. How-
was assumed to remain constant until the year ever, when an adapted model is used, the user
2000, and in the other an annual growth rate of needs to know what the model was originally in-
three percent was assumed. The model pre- tended to do, what its new purpose is, what
dicted that in the case of the three percent per changes have been made, and whether these
annum increase in gasoline price, new car sales have been done correctly.
would be about twenty-five percent less in the
year 2000 than if the gasoline price remained ///ustration
constant (Jack Faucett Associates 1976). This Several examples drawn from uses of
demonstrates how sensitive output may be to a automobile demand models can serve to illus-
change in an exogenous input that may be trate what can occur. Both of the major
regarded as small given today's rapidly increas- econometric models of automobile demand
ing gasoline prices. (Faucett and Wharton EFA models) in use for
federal policy analysis were created primarily to
Significance evaluate impacts of energy policies such as
This illustration demonstrates the influence mandatory fuel economy standards and various
that exogenous variables can have on model tax, rebate, and penalty policies related to fuel
output. It is important to realize that the validity economy and fuel consumption. Attempts have
of the model output is directly dependent upon been made to use one or the other of these
the validity of the exogenous input. This re- models to investigate policy in other areas. In
quires a careful examination of input data as general, these attempts either have been un-
part of establishing the utility of the model out- successful or have worked only after additions
put in policy analysis. or revisions were made to the models.
An attempt was made in the Department of
Transportation to use the Wharton €FA auto
model to evaluate the impact on fuel consump-
tion of the fifty-five-mile-per-hour speed limit.
This failed because there was no way in the
model to separate the changes in fuel
consumption achieved by lowering the speed
limit from those resulting from use of more fuel-
efficient vehicles or from changes in the num-
ber of miles people drove.
The Wharton €FA auto model has been
adapted to analyze nonenergy policies, such as
vehicle emission control standards and passive
safety restraints. In these cases other models to
estimate costs of controls or restraints had to Limitation 7. The Apparent Precision of
be built or run to provide the Wharton EFA Model Forecasts may be Misleading
model with necessary cost input data. The
model has also been adapted to forecast the
Model output appears very precise. It is given
impact of battery-powered automobiles. In this
as a specific number with several sig'nificant
case a special submodel had to be built to
digits. That does not mean it is accurate or
generate the cost factors related to electric
reliable. If a model is built using statistical tech-
vehicles, and the main model had to be re-
niques, as econometric models are, a con-
structured to account for the new battery-
fidence band can be constructed that gives a
powered cars.
range around the numerical result within which
the true value can be expected to lie with a cer-
tain probability, if the appropriate assumptions
hold. It is similar to saying that a falling satellite
Frequently a user may not know will, with a 90% probability, hit a certain latitude
how accurate model results are. plus or minus a few degrees.
Confidence bands can easily be calculated
for small models. For very complex models, like
the ones of interest in the transportation policy
Significance area, simple analytical techniques do not yet
Because models are designed for limited exist to calculate precise confidence limits.
purposes, attempts to use them for what may Relatively sophisticated techniques are avail-
seem to be logical extensions can lead to able to calculate confidence bands for fore-
failure. Models are adaptable, but adaptations casts of larger models, but are expensive and
frequently mean that additions have to be made not widely used. Frequently therefore, a user
to the model and structural changes have to be may not know how accurate model results are.
made inside the model. When changes are In policy studies, models are used to study
made inside the model, total model per- the differences between impacts of alternate
formance may be affected because the parts proposed policies. Because the accuracy of
are so interconnected. The results of previous model output is uncertain, it is difficult to know
sensitivity tests may no longer be a guide to how when a difference between two forecasts is
the model performs, and new tests will be re- large enough to be meaningful. Failure to
quired. recognize this can lead to unsound conclusions
Add-on submodels are a common way to about which alternative is likely to produce the
modify models and increase their utility. How- desired results.
ever, when an add-on model is created to pro-
vide new input, the accuracy of the main model lllus tra tion A
output becomes dependent on the accuracy of Two illustrations of this problem are
the output from the add-on. presented here. The first deals with the con-
The obvious significance with regard to fidence band on the forecast values of the
policy analysis hinges on whether the model vehicle-miles-of-travel-per-household (VMT)
"blows up" or produces output. If it does not equation of the Faucett Automobile Sector
work, the user has a clear indication that some- Forecasting Model. Values of this variable were
thing is wrong. However, if output is produced, calculated for 1977, 1978, 1979, and 1980. The
the user may be unaware that it is useless, and values of these were approximately 15544,
policy recommendations may be made based 15956, 16298, and 16637, respectively. These
on the output. values seem to be of reasonable magnitude and
to exhibit a reasonable growth pattern. How-
ever, the confidence band for each of the values
was calculated to be approximately f1000, at
the ninety-five percent level of confidence.

Significance
The consequences of relying on output data
with such confidence limits are obvious. The
confidence bands in this case are so large that theoretical difficulties may preclude the
there may not be true differences among the computation of statistical confidence intervals
year to year values of the VMT estimates. A for the predictions of large models such as
policy based on this type of information may be Wharton EFA's, less rigorous estimates of pre-
unsound. Yet, unless such a lack of precision diction precision would be a great aid to deci-
were made explicit, policymakers could easily sion makers.
be misled. One can only speculate what the outcomes
would have been if each "number" had been
lllustration 6 properly qualified. The differences in inter-
A second illustration deals with two applica- pretation of similar numbers highlight the need
tions of the Wharton EFA Automobile Demand to properly qualify results and to inform policy-
Model by two different sets of analysts. makers of the uncertainty of forecasted values.
The National Highway Traffic Safety Adminis-
tration (NHTSA) made prominent use of the
Wharton EFA Auto Demand Model in the
documentation supporting the fuel economy
standards for automobiles for 1980-1984.
NHTSA reported that the proposed 27 miles per
gallon standard for 1984 would lead to only
210,000 fewer new car sales than the 1980 stan-
dard of 20 miles per gallon. The difference was
1.8 percent of the forecast 1984 sales. NHTSA
labeled this difference as "insignificant, given
the difficulties of projecting the sales initially"
(National Highway Traffic Safety Administration
1977).
In contrast, analysts of the International
Trade Commission (ITC) used the same model
in conducting a study for the Senate Finance
Committee of the proposed "gas guzzler" tax.
This study projected a shift of 300,000 in sales
from domestic to foreign producers in 1985 if
the tax and rebate plan were enacted ( U S .
International Trade Commission 1977). This
represented a shift of slightly more than two
percent of total sales. Senate Finance Com-
mittee staff members report this was viewed as
significant and it contributed to the delay in ac-
tion on the gas guzzler tax proposal.

Significance
Both of the projected differences are of about
the same size. One group determined the figure
to be significant while the other labeled the
amount insignificant. In both cases, the judg-
ments of significance and insignificance were
subjective. Both the magnitude of the estimate
(number of cars), and the degree of precision of
the estimate judged to be significant depend on
the different perspectives of NHTSA and the
ITC in the context of specific problems. In this
case, however, it is not clear that these judg-
ments were derived from an adequate under-
standing of the uncertainty of the forecasts
since confidence intervals were not associated
with the predictions. While practical and
 Questions a Policymaker Should Ask
Before Using a Model

Given the complexity of models used in the policymaking

arena and the limitations inherent in their use, a policymaker
should obtain answers to several questions before making any
decisions based on the output of a model. Several such ques-
tions are discussed here.

How well does the model Assuming that an analyst has chosen a particular model for
perform? use in a particular policy-related application, the policymaker
should check on the quality of its performance. There are
three ways of doing this if the model is based on historical
data: first, by examining the model's output over the sample
period and comparing it with observed data for that period;
second, by examining the model's output for the time period
starting just after the fit period of the model through the pre-
sent and comparing it with actual data for that period, if they
are available; and third, by examining the model's output for
future years and checking for its "reasonableness."
Of these three, the second alternative is probably the best
way of checking the model's "track record." It affords the
opportunity to test the model in a forecasting mode, yet it of-
fers the advantage of having historical data available to com-
pare with the output. Note, however, that this method will be
less useful if only short-term data are available when a long-
term model is being tested.
Testing over the historical period is sometimes neither
feasible nor appropriate because of the nature of the model,
its method of construction, or other factors. Testing the model
over the future may not yield adequate information to make
the decision about its accuracy, since sometimes it is impossi-
ble to judge whether the output is reasonable. There simply
may be no basis for comparisons.

- - -

Has the model been analyzed by Often in the course of building a model, the author will per-
someone other than the model form various tests in an attempt to validate the model. These
authors? test results, if they include model output and are objective, can
probably be viewed with some confidence. However,
modelers often do not take the time to rigorously analyze or
assess their models themselves, primarily because the time
and resources allocated to model building are limited. Con-
tracts requiring model construction usually do not include a
separate task for model analysis.
Model validation tests performed after a model has been
constructed give little insight into the theory and dynamics of
the model. For a user to have an understanding of the model,
he should have access not only to the model documentation
but to any assessments performed by people other than the
model builders. Such assessments can provide insight into
 the strengths and weaknesses of a model and provide a more
objective view of the model than may be provided by a model
author. The results of such an assessment should be carefully
reviewed and taken into account before a model is chosen for
use in policy-related studies. It should be noted, however, that
model assessments are not often performed.

Is adequate documentation of Adequate documentation is crucial if persons other than the

the model available for all who model builders are to understand it. The documentation
wish to study it? should describe the method of model construction, the theory
behind it, the data and assumptions used, the equations, the
method of operation, the input required for running the model,
and sample output. The user should have enough information
available to evaluate the reasonableness of the assumptions
used in constructing the model. The documentation should
make clear what parts of the model have been based on
historical fact and what assumptions have been made.

What assumptions and data The assumptions and data used in running a model for
were used in producing model specific applications are generally different from those used in
output for specific applications? constructing the model. In running econometric models to
produce projections, a set of exogenous data consisting of
forecasts of several variables is generally required as input.
These input data are themselves forecasts of the unknown
future and should be used only with care and an under-
standing of their limitations.
If a model has already been run for a specific purpose, the
set of assumptions and data used to produce the output
should be known, so that their reasonableness and ap-
plicability can be determined.

Why is the selected model There may be many models that are suitable, at least based
appropriate to use in a given on initial inspection, for use in a particular situation. It is up to
application? the policymakers to satisfy themselves that the most ap-
propriate model has been selected. Questions that should be
asked include: What is the stated purpose of the model
selected? What does it measure? What does it not measure?
Is its intended use compatible with the present need? Is this
the easiest model to run that is applicable to the study area of
interest? Are there other models equally suited to the job?
Finding the answers to these questions may be a very time-
consuming effort. However, it is advisable to have the answers
in hand so that resources may be most effectively used. Many
models may forecast the same variables, but some may do
more. If two models are of equal quality (which is difficult to
determine) and a user is interested only in the output of the
less complex model, clearly it would be wasteful to run the
more complex model.
It should also be determined that the model chosen for use
actually forecasts the variables of interest and that they are not
buried somewhere internally in the model, or worse, set ex-
ogenously. Often this distinction is not clear.
 - - - -

Was the model run directly and A given model may be run by a number of users for a variety
specifically for the present of purposes. It may be that for one of those past uses, the
purpose? model input and output seem similar to those desired for a
present policy analysis. Extreme caution should be exercised
if output from other applications is used. Caution must also be
exercised when individuals in other agencies perform model
runs on request for a specific application. One can never be
sure of the exact circumstances under which a model was run.
Input may not coincide directly with current needs. Alternative
options in programs may sometimes be exercised. Biases in
interpretation of the meaning of output may exist. If a model is
not run by a user for a particular policy application, the
chances of errors appearing in the analysis are greatly in-
creased.

What is the accuracy of the Many models have output that is accurate only within some
model output? error band. The larger the error band at some level of con-
fidence, the less accurate the output. It is relatively straight-
forward to determine confidence bands for small, single-
equation models, but more difficult for large-scale models.
Nevertheless, it is imperative that the model user have some
idea of the accuracy of the model output before it is used in
specific applications. In comparing output of a model run that
uses two different sets of input data, the error bands on the
output may be so large that results that look different may not
be, in a statistical sense. Knowing the accuracy of the output
helps to put the usefulness of the model into perspective.

- --- .

Does the structure of the model A model is an abstraction of reality. In translating from
resemble the system being reality to mathematical equations, some components of the
modeled? real-world system are omitted. It is important to identify which,
if any, pivotal elements of the real system have not been in-
cluded in the model. Key items and relationships included and
the key ones omitted should be identified. In addition, while an
attempt may be made to include in the model some aspect of
the real-world system, its representation in the form of an
equation may be inappropriate or inadequate. The bases of
the mathematical representation should be clear to the model
user.

Is the model appropriately A model is constructed to represent a real-world system

sensitive to the inputs being and predict the response of that system to changes made to it.
varied? The latter is referred to as the sensitivity of the model. Even
though a model was constructed so that its output would vary
appropriately with changes in its input, this does not always
occur.
It is the responsibility of the model user to ascertain, either
through review of the model documentation or assessments of
the model, whether the model is appropriately sensitive to
changes in the input variables of interest. If it is not, the out-
put of the model may indeed be useless for the intended pur-
poses.
13
 Summary and Conclusions

Mathematical models are in widespread use several questions relating to model use. These
in policy analyses related to the transportation include queries concerning the model's perfor-
system. There are many kinds of mathematical mance record, results of model assessment, the
models, with econometric models being the purpose of the model, its appropriateness in
primary kind used in the motor vehicle specified applications, assumptions contained
transportation policy sector. i n the model, and availability of model
While mathematical models may provide documentation. Analysts who use models to
policy analysts with strong tools to use in their formulate or analyze policies have an obliga-
studies, they may also provide very misleading tion to answer such questions. These answers
results if not applied correctly. There are many should be public so that peers can review their
limitations in the correct use of models. Some reasonableness.
limitations are inherent in a model (e.g., models The proper use of models can add con-
are incomplete, and model output is uncertain siderable insight to the policymaking process,
although it may appear precise). Other limita- but model output should be regarded only as
tions arise when models are used (e.g., the ac- approximations. Only if policymakers are aware
curacy of input data may be unknown, and the of the limitations inherent in models can
operational status of a model is often unclear). mathematical modeling enhance the policy-
To help ensure proper use of models in making process.
policy analyses, a policymaker should ask

Acknowledgment

Development of this independent paper was supported by an un-

restricted grant from the Motor Vehicle Manufacturers Association. The
assistance and advice of our colleagues, particularly W. S. Barnett, D. H.
Hill, D. C. Roberts, P. B. Sanghvi, L. D. Segel, and D. B. Suits, are
gratefully acknowledged.
 References

Difiglio, C., and Kulash, D. 1976. Marketing and mobility: Report of a

panel of the interagency task force on motor vehicle goals beyond 1980.
Interim report. Washington, D.C.: Federal Energy Administration.
Energy Information Administration, U.S. Department of Energy. 1978.
Annual report to Congress: Volume 11 1977. Washington, D.C.: U.S.
Department of Energy report DOEIEIA-003612.
Fromm, G.; Hamilton, W. L.; and Hamilton, D. E. 1974. Federally sup-
ported mathematical models: Survey and analysis. Washington, D.C.:
U.S. Government Printing Office.
Golomb, D. H.; Luckey, M. M.; Saalberg, J. H.; Richardson, B. C.; and
Joscelyn, K. B. (pending). An analysis of the Wharton €FA automobile
demand model. Ann Arbor, Michigan: UMI Research Press, an imprint
of University Microfilms International (in press).
Jack Faucett Associates, Inc. 1976. Fuel economy policies and their ef-
fects on automobile ownership, use, and fuel consumption: Final report.
Report submitted to the Federal Energy Administration, Washington,
D.C.
Kelderman, J. 1978. DOE claims makers can't meet '80s MPG. In
Automotive News May 15, 1978. Detroit. P. 1.
National Highway Traffic Safety Administration. 1977. Final impact
assessment of the automotive fuel economy standards for model year
1981-84 passenger cars. U S . Department of Transportation report
DOT-HS-803-183.
Schink, G. R., and Loxley, C. J. 1977. An analysis of the automotive
market: Modeling the long-run determinants of the demand for
automobiles. Volume I:The Wharton €FA automobile demand model.
Volume 11: Simulation analysis using the Wharton €FA automobile de-
mand model. Volume 111: Appendices to the Wharton €FA automobile
demand model. Transportation Systems Center final report DOT-TSC-
1072.
Shonka, D. 6.; Loebl, A. S.; and Patterson, P. D. 1977. Transportation
energy conservation data book: Edition 2. Oak Ridge, Tennessee: Oak
Ridge National Laboratory.
U.S, International Trade Commission. 1977. The fuelefficiencyincentive
tax proposal: Its impact upon the future of the U.S. passenger auto-
mobile industry. Report to the Committee on Finance of the United
States Senate. Washington, D.C.: U.S. Government Printing Office.
 Other Titles of Interest

An Inventory of Selected Federal Policy Applications of An Analysis of the Wharton

Mathematical Models Relating the Wharton EFA Automobile EFA Automobile Demand Model
to the Motor Vehicle Demand Model
Transportation System and D. Henry G o l o m b ,
Associated Literature James H. Saalberg, Michael M. Luckey,
Barbara C. Richardson, James H. Saalberg,
Barbara C. Richardson, Kent B. Joscelyn Barbara C. Richardson,
L a w r e n c e D. Segel, K e n t B. Joscelyn
W. Steven Barnett,
/
The Wharton EFA Automobile De-
Kent B. Joscelyn mand Model is one of the more prom- The Wharton EFA Automobile De-
inent analytical tools ever deve'loped mand Model was introduced in 1977
for use in policy analyses related to as an analytical tool to support policy
4 This one-volume reference contains the m o t o r vehicle transportation
descriptions of selected mathematical making related t o the automobile
system. transportation system. The model has
models (econometric, physical, ac- This study identifies the agencies
counting, etc.,) relating to the motor been used to support rulemaking and
that have used the model, determines
vehicle transportation system and to analyze the effects of various gov-
what p o l i c y issues have been
abstracts of associated documents evaluated using it, investigates how ernmental policies. In general, these
that relate to models and the policies the model has been used, and dis- applications occurred without a criti-
analyzed by the models. cusses how effectively and appro- cal examination of the model.
Complete references and sum- priately the model has been applied. This detailed analysis reveals that
maries are given for the models and This detailed case study describes while the model does contain several
associated literature. Descriptions in- how and by whom the model has been innovations in the forecast of auto
dicate model objectives, limitations, applied in policy-related analyses demand, it has serious shortcomings.
and benefits, structure, data and com- from its first known use in the spring Trends and levels of such critical
puter requirements, as well as other of 1977 to March of 1979. variables as new car sales by size
relevant information. Few analysts who used the Whar- class are not predicted well.
The study's indices make it possible ton EFA model results appear to have This first thoroug h evaluation of the
t o identify models and documents ac- had a deep understanding of the model provides not only a complete
cording to model name, report title, model, including some of its signifi- analysis of a currently prominent ana-
cant limitations. This is especially lytical tool, but it also presents an ap-
keywords, personal and organiza-
true for analysts who did not directly proach that can be used to evaluate
tional authors, and sponsors.
operate the model. Lack of familiarity other models. The text is essential for
Indispensable to anyone in the au-
with details of the model appears re- those who may consider use of the
tomotive transportation field, this ref- lated to inappropriate and inade-
erence covers a wide variety of subject Wharton EFA Automobile Demand
quately qualified prominence being
areas including automobile demand, Model or its output. It is also of great
accorded to some model results. An
fuel consumption and economy, air value of those who build or evaluate
important conclusion is that policy
pollution, market share, and vehicle models.
analysts and decision makers need to
miles traveled. understand better the nature and limi-
This represents the first volume in tations of the models they employ. ISBN 0-8357-1083-1 $14.95
a series. Additional volumes will be The study findings will be of inter- 302 pages
issued on an annual basis. est t o those concerned with the au-
tomotive transportation system and
ISBN 0-8357-1084-X $17.50 modeling. Policymakers and
394 pages modelers will benefit from an exam-
ination of this instructive case history.
Faculty and students will find the re-
port valuable as instructional material.

ISBN 0-8357-1082-3 $7.50

68 pages