T e c h n i c a l pa p e r

A Military Effectiveness Analysis

and Decision Making Framework for
Naval Ship Design and Acquisition
John C. Hootman and Dr. Cliff Whitcomb

Abstract Nomenclature
This paper describes research that develops a new as combat effectiveness, survivability, and initial DPs = Dimensional Parameters
framework for performing military effectiveness cost may be equally important in the design MOPs = Measures of
analyses and design tradeoff decisions. It provides an process. (Rains 1984) Performance
MOEs = Measures of
extensive survey of literature for effectiveness analysis Effectiveness
and multi-criteria decision making to develop a single While it is clear that survivability and cost have MOFEs = Measures of Force
consistent philosophy for such analyses. become central issues in naval ship design and Effectiveness
OMOE = Overall Measure
This philosophy is applied to a requirements and acquisition programs, it is not as clear that ef- of Effectiveness
effectiveness analysis case study of a conventional fectiveness analysis is playing as large a role as it MOM = Measures of Merit
submarine that is performed using response surface can and should. A vital component of the design AoA = Analysis of Alternatives
methods to facilitate design space visualization and MoRs = Military Operations
of these systems is the ability to measure these Research Society
decision maker interaction. Measures of merit are
effectiveness characteristics, which is a difficult MCDM = Multi-Criteria
developed and applied to the case study. The result-
task. As Zink et al. observe: Decision Making
ing requirements space and methods to visualize and WS = weighted sum
explore it in a decision making context are presented HWS = hierarchical
and discussed. Lastly, a framework is proposed that Measures and targets that [drive] these stud- weighted sum
would facilitate the concurrent consideration of re- ies are dependent on the subjective opinion of AHP = analytical hierarchy
process
quirements and effectiveness analyses with design and the customer/user, i.e. the requirements. These
MAU = multi-attribute utility
technology forecasting to create a unified tradeoff en- requirements are often ambiguous and typically RDM = rational decision
vironment that would provide decision makers with change over time. Therefore, understanding the making
pertinent information to facilitate better informed re- RSM = Response Surface
simultaneous impact of requirements, product
Methods
quirements derivation and design selection. design variables, and emerging technologies RSE = Response Surface
during the concept formulation and develop- Equation
Introduction SSK = conventional
ment stages is critically important, and until now
(non-nuclear) submarine
The design of an effective system rests upon elusive. (Zink et al. 2001) STS-EB = Survivability of
understanding how to measure system effective- Suspected Target Search
at the End of Burst
ness, how to draw an appropriate boundary to During the first half of the Cold War, “ship level
STS-ES = Survivability of
define the extent of the system to include in the requirements, rather than the ship’s contribution to Suspected Target Search
analysis, how to clearly and accurately represent the performance of the task force, drove the design at the End of Search
PDS = positive Detection swath
this and other design information to decision- process”. (Rains 1999) Beginning in the late 1970s
AIP = Air Independent
makers, and how to make rational design deci- and early 1980s naval engineers realized that it was Propulsion
sions. Dr. Dean Rains, one of the most prolific important to look at the collective whole of how a W = Width of PDSW
V = Speed of Searcher
authors on the subject of military effectiveness vehicle or weapon was assembled, which led to the
Vmax = Burst Speed
analysis for naval ship design notes that: use of systems engineering concepts in a naval sys- VEES = STS Evasion Endurance
tems context, the total ship system. But now in an Speed
Combatant ship design is a series of tradeoffs era of jointness, network centric warfare, and cost tB = Time at Burst Speed
t = Time of Search
often made with little knowledge of the impact constraints, we must look beyond the total ship
tO = Time Late
of the decisions, except on ship size or displace- system to the battleforce; engineers must consider UTE = Unified Tradeoff
ment. However, many other considerations, such how the system that they are designing interacts Environment

Naval Engineers Journal SUMMER 2005 43

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

with the environment it operates in and the other and make critical trade-offs that proceed the
systems it operates with. formalization of requirements.” (GAO 2001)
Therefore, Frits et al. advocate a shift of design
This expansion of scope was coined the super- philosophies that would lead to the develop-
system, which includes everything outside the ment of:
ship that either affects it or is affected by it. To
evaluate systems in the supersystem context, an environment in which the effects of changes
appropriate metrics must be applied. These are in engineering parameters are analyzed to
generally called measures of merit and they are determine their impact on overall...effective-
generally considered to be “inherent in the mis- ness. This process is accomplished by linking a
sion and are external to the ship”. (Hockberger conceptual...design program with a [simulation]
1996) goes further to stress the importance of program. Thus, the linkages between design
evaluating effectiveness in a mission context: variables, weapon performance, and tactics can
be more thoroughly understood, and a vehicle
The ship’s effectiveness has to do with the with the greatest overall effectiveness can be cre-
change in the military situation that results ated. (Frits et al. 2002)
from its involvement in the engagement,
which is a matter of outcomes, and measures Such concurrent development of effectiveness
of effectiveness can thus be seen as outputs of models and engineering analysis is required to
engagement...[thus] it is the synergism between optimize a system and provide decision makers
the new ship or system and the rest of the task with pertinent information to facilitate better
force that is at issue, and it is the task force ef- informed requirements derivation.
fectiveness and attainment of mission measures
of effectiveness that must be used as the basis This paper will take the first steps to develop-
for assessing and comparing the performance of ing such a system by examining a case study for
each alternative. (Hockberger 1996) a conventionally powered submarine. In order
to gain a firm understanding of the simultane-
In the case of torpedo design research, (Frits et ous impacts that Zink et al. describe, the ship
al. 2002) observed that the use of effectiveness designer must be introduced to subjects that have
analysis existed, but it was virtually decoupled traditionally been beyond the designer’s purview.
from the design process. The analysis appeared Further, to design a modern, highly complex
in series with the design work, leading to an it- engineering system, the designer must understand
erative cycle in which Fleet operators developed what external factors are most important to the
torpedo tactics, had a torpedo built, and then design, the interaction of these multiple, com-
re-developed tactics to better suit the torpedo peting design factors, how the system relates to
that was delivered. its environment, and frameworks that decision
makers use to evaluate the system. Therefore, this
A similar perspective was echoed in a Govern- discussion will first synthesize competing theories
ment Accounting Office report on best practic- of systems effectiveness analysis and multi-cri-
es in weapon systems procurement. It dem- teria decision making into consistent principles
onstrated that the current practice of setting to approach the problem of requirements and
requirements prior to the designation of funds effectiveness analysis for naval ship design. These
to conduct systems engineering denies deci- principles will then be applied to a requirements
sion makers and designers of “the knowledge and effectiveness analysis for the design of a con-
needed to match wants with resources before ventional submarine. The results of applying this
starting a program...to evaluate the sufficiency tradeoff methodology will be presented, and the
of available resources — knowledge, time, discussion will wrap up with important conclu-
money, and capacity...in time to help identify sions and recommendations for future work.

44 SUMMER 2005 Naval Engineers Journal

Measures of Performance, effectiveness (MOSEs), or as an overall measure
Effectiveness, and Merit of effectiveness (OMOE).
An extensive literature survey was performed on
the subject of measures of merit (MOMs). This Measures of Merit (MOMs) — MOMs are a
section will summarize that review and develop general term for all measures that character-
a single, consistent description of a MOMs ize a system under analysis, they “subsume all
system for application to ship concept design, a measures that characterize a...system” (Green
so-called MOM philosophy. and Johnson 2002). In this study, MOMs will
collectively refer to MOPs, MOEs, and MOFEs.
Definitions
While there is no consensus on specific defini- As the definitions indicate, MOMs develop in a
tions, the following definitions appear to be very hierarchical manner.
broadly accepted and will serve as the baseline
for this work: Measures of Merit
After defining the key terms used to describe
Effectiveness — “Effectiveness is the condition MOMs, the varying theories of what constitutes a
of achieving a requirement” (Hockberger 1996). MOP or MOE were assessed. The most struc-
tured and significant work towards a unified the-
Dimensional Parameters (DPs) — “DP s are the ory of MOMs appears to be from weapons and
properties or characteristics of the physical enti- combat systems designers, principally the Military
ties whose values determine system behavior and Operations Research Society. One of the most
the structure under consideration even when at prolific authors from this constituency is Green,
rest”. (Green and Johnson 2002). who discusses the importance of bounding the
system in terms of internal and external attributes
Measures of Performance (MOPs) — MOPs early in the process of developing MOMs. This
are “related to inherent parameters (physi- is a crucial and often overlooked step because “a
cal and structural) but measure attributes of change in the boundaries changes the parameter
system behavior”. (Green and Johnson 2002). set and the resulting system behavior and perfor-
MOPs are generally “non-probabilistic mea- mance”. (Green 2001a)
sures of performance, where ‘the MOP class
provides for the collection of metrics...that A useful method to visualize this is a series of
are not probabilities of successful outcomes of concentric rings, similar to a sliced onion or tree,
functions.’ Thus MOPs are the ‘consequence’ as shown in Figure 1.
of specific configurations of physical ele- Figure 1: System
ments.” (Brown 1995) Boundary Levels (Green
and Johnson 2002)

Measures of Effectiveness (MOEs) — MOEs are

a “measure of how the system performs its func-
tions within an operational environment” (Green
and Johnson 2002). MOEs are metrics that
measure “the degree of effectiveness attained in a
achieving a requirement” (Hockberger 1996).

Measures of Force Effectiveness (MOFEs)

— MOFEs are a “measure of how the system,
and the force of which it is a part, performs its
missions” (Green and Johnson 2002). MOFEs
may also be referred to as measures of system

Naval Engineers Journal SUMMER 2005 45

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

Green begins by specifying the DPs and MOPs as 2001a) The Air Force AoA guidebook advises
characteristics that are measured within sub- that “MOEs should not be strongly correlated
systems and the system, “whereas MOEs and with one another (to avoid overemphasizing
MOFEs are specified and measured external to particular aspects of the alternatives)...[and that]
the [system] boundary” in relation to associated MOEs must be independent of the nature of the
forces or environments. (Green and Johnson alternatives, as all alternatives are evaluated us-
2002) In discussing models used for effective- ing all MOEs” (OAS 2000).
ness analysis (Leite and Mensh 1999) specify
two groups of metrics, similar to Green’s system Green proposes that the result of such an ap-
boundary levels: those related to the model and proach is a balance “between those elements,
its internal operation, and MOMs for the “system both combat systems and ship systems, that are
performance as a function of its intended opera- required for mission success [and that the] pro-
tional employment”. (Leite and Mensh 1999) cess model focuses on the mission goals rather
than starting with a set of constraints that accept
Green describes a process model that begins with degradation in the performance of these goals as
four inputs: the mission, the expected threat, the a price that must be paid” (Green 2001b)
environment, and potential system concepts. In
the case where a ship is the system under analy- Green also advocates that “expressing MOPs,
sis, Green recommends “viewing the ship as a MOEs, and MOSEs as a probability allows us
weapons system [to keep] these performance to determine if a parametric change is statisti-
goals in context with the assigned missions”. cally significant” (Green 2001a)) Further, Green
(Green 2001b) This implies that MOEs should insists that the MOMs developed for use in
be developed in parallel with the system require- analyses must be “efficient in the statistical sense
ments, and Hockberger stresses that this needs to (small variance/reasonable accuracy).” (Green
be done because: it can be done, they help formu- and Johnson 2002) Lastly, Green concludes
late requirements, and it helps make the design with the advice that “if it can’t be expressed as
process more efficient. (Hockberger 1996) a probability it probably is not an effectiveness
measure.” (Green 2001a)
To be able to conduct such a mission analysis, a
model of the system under development and its While there is no “magic list of canned effective-
warfighting environment must be developed. After ness measures” (Green 2001a) for early stage
developing an appropriate system model, the out- development, there have been many studies
puts of the scenario are used as inputs to metrics performed in the past, and many examples of
for representing the previously defined MOMs. MOMs can be drawn from these. These exam-
ples can either be applied directly to the problem
Green keeps the focus of the work on mission at hand, or serve as a springboard for develop-
and system solutions by relating “operational ing more appropriate MOMs. For example, the
availability, reliability, survivability, and weap- Mine Warfare Center uses 28 MOPs with four
ons systems performance...to their subsequent functional categories (sense, engage, control, and
impact on ship design”. (Green 2001b) Similar logistics) that were chosen to be applicable to
to Green, Brown develops a MOM hierarchy all of their mine countermeasures studies (Mine
from a cycle of mission accomplishment’ com- Warfare Center A-2G-2758). More specifically,
posed of: availability, reliability, survivability, Liete and Mensh listed many successful MOMs
and capability. (Brown 1995) from their work and experience, and these are
summarized in Table 1.
While developing the MOMs, the literature
stresses that the measures “must be independent It is possible to have multiple MOEs, and even
at the level of analysis under evaluation”. (Green MOFEs (hereafter called top-level MOMs).

46 SUMMER 2005 Naval Engineers Journal

Thus, it is improper to use the term ‘optimal’ too
Table 1 Sample MOMs [Leite and Mensh, 1999])
loosely, because the optimization of multiple,
competing attributes is a much more difficult DPs MOPs MOEs
problem than that of the optimization of a single Size Gain Probability of detection
attribute. Therefore, when multiple top-level Weight Throughput Reaction time
MOMs are in use, multi-criteria decision making Aperture size Error rate Targets designated
methods must be used to accurately and objec- Capacity Signal to noise ratio Probability of kill
tively model and determine system effectiveness. Location/orientation Fragment size/pattern
Firing arcs/cutouts
Lastly, a brief note on cost must be made. Cost
should be excluded from a military effective- the most prevalent, to include: weighted sum
ness analysis; however, it cannot be excluded (WS), hierarchical weighted sum (HWS), analyti-
from the complete design tradeoff analysis. As cal hierarchy process (AHP), multi-attribute util-
dictated by DoD’s cost as an independent vari- ity (MAU) analysis. Unfortunately, this discussion
able policy, it should be an independent consid- can only introduce these methods; for a more
eration, treated as any other top-level tradeoff in thorough discussion of the above MCDM models
the design process. with respect to ship design consult (Whitcomb
1998a) and with respect to complex systems in
Multi-Criteria Decision Making general consult (deNeufville 1990).
The determination of an optimized design is not
one that can be approached from traditional, The WS method is the simplest, and most com-
analytical optimization methods such as objective monly implemented of the methods to be dis-
function definition and use of different gradient cussed. This method is implemented by summing
methods, knee-of-the-curve, or Kuhn-Tucker nec- the product of objective weights and attribute
essary and sufficient conditions. Rather, the pres- levels (MOEs in effectiveness analysis) to arrive
ence of multiple criteria must be considered and at a figure of merit. (Whitcomb 1998a) This
multi-criteria decision making (MCDM) methods method is overly simplistic and insufficient for
must be used. This can lead to the determinations anything except the most basic of investigations.
of many optimums requiring the use of Pareto
analysis and rational decision making. The remaining three MCDM models are more
complex and are similar in that they are all
Further, design decisions of these types are based on a hierarchical approach, somewhat
rarely made unilaterally, so decision processes analogous to the discussion on MOMs. This ap-
in teams must be examined. This examination proach eliminates the first concern with the WS
has the potential to lack some of the mathemat- model, and greatly aides in realizing the second
ical rigor that MOM development demonstrat- concern. According to Whitcomb, three major
ed because it more directly involves multiple advantages of the use of hierarchical relation-
stakeholder interaction and conflicting prefer- ships are that they (Whitcomb 1998a):
ences. Because stakeholders draw knowledge ■ Refine the ability to define appropriate aspects
from personal experience, and preferences, it is of each MOE.
very important that any preferred methodology ■ Show objective function relationships to each
that is identified be internally consistent and other.
rational to prevent natural biases from skewing
■ Organize the evaluation.
the MCDM process.
The simplest model that uses a hierarchy is the
There are many methods that can be used to hierarchical weighted sum. This method is a
model MCDM, but this work will introduce only “modification of the weighted sum method, us-

Naval Engineers Journal SUMMER 2005 47

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

ing the objective hierarchy versus the single level ity to incorporate the decision maker’s nonlinear
objective sum of products formulation” that the preferences towards each of the objectives into
WS method used. (Whitcomb 1998a) A byprod- the decision process”. (Whitcomb 1998a)
uct of the straightforward nature of this method
is its ease of use and easy implementation with Aggregation in general is not a bad solution to
spreadsheet models. simplify MCDM problems. As the foregoing
discussion has shown, the aggregation of lower
The analytical hierarchy process is similar to levels of the hierarchy is vitally important to
the HWS, except it reflects customer or decision most of the methods. The Air Force AoA Guide-
maker preferences and priorities. (Saaty 1988) book refers to aggregation of MOMs as rolling
The key to this method is the use of pairwise up the results, which allows decision makers to
comparisons of every attribute at each level of the compare the alternatives with a smaller number
hierarchy. By performing these pairwise compari- of measures; however, the “advantage of having
sons, a relative importance scale is developed for a smaller number of measures carries the obvi-
each attribute. Whitcomb notes that a benefit of ous disadvantage: information, and along with
this method is that it inherently provides a consis- it potential insight, is lost in the roll up process”
tency check of the pairwise comparisons. How- (OAS 2000) They propose only using aggrega-
ever, as the number of attributes under consider- tion when it is firmly grounded in sound logic
ation “becomes large, approximately greater than and meets the following conditions (OAS 2000):
seven, decision makers may have trouble keeping ■ The aggregation arises naturally from relation-
the criteria straight”. (Whitcomb 1998a) ships among the MOEs

The final major MCDM model to be discussed ■ The significance of the aggregates is clear
is multi-attribute utility analysis, which is almost ■ The aggregates tell a clearer story than the
solely grounded in customer or decision-maker individual MOEs
preferences and priorities; however, it also
includes other characteristics such as uncertainty Further, DODI 5000.2 warns against methods
and risk (Keeny and Raiffa 1976). Whitcomb that lead to customer or preferential weighting
notes that the MAU analysis does not directly of different attributes:
use the hierarchy developed earlier, but it can
play a vital role in ensuring the independence of Never use schemes in which several measures of
the attribute in the analysis. effectiveness are weighted and combined into an
overall score. Weighting schemes are sometimes
This model is based on the utility function, helpful, but they must be clearly explained in the
which is “a specific type of value function in that analysis so that their results can be interpreted
the units are based on an ordered metric scale correctly. (Brown 1995)
and is developed under the condition of risk.”
(Whitcomb 1998b) Because complex decisions It is interesting to notice the contradiction be-
have numerous attributes, this method combines tween the official guidance and the more mature
the individual utilities into a single function, the and useful methods of MCDM that all involve
MAU function. These are analytic functions, some form of weighting. If weighting is avoided,
thus “the use of an ordered metric scale allows then the decision-maker will be presented with
utility to be defined with respect to any two much more information than they either want or
points on the scale, which are then assigned any can be reasonably expected to handle, or both.
convenient value. The quantities for the worst Therefore, perhaps Hockberger’s comments on
and best decision outcomes can be defined, the subject strike a reasonable compromise:
forming the basis for actual measurement of util-
ity” (Whitcomb 1998b). In practical application, Lower level MOEs should be calculated and
a major benefit of the MAU method is “the abil- combined within the model or simulation, which

48 SUMMER 2005 Naval Engineers Journal

can determine the way each MOP of an alter-
native concept contributes to achieving them
and how they combine to produce higher level
MOEs. Human judgment and weights are only
required for going the rest of the way up the
tree, combining the MOEs the model yields in
order to produce the overall composite MOE.
(Hockberger 1996)

This compromise still leaves the decision

maker with the task of performing one or more
tradeoffs, but at least of far fewer competing
options. A widely accepted method for visual-
izing these various alternatives in relation to one
another is the Pareto plot.
Figure 2:
To generate the Pareto plot, the decision maker tier and are thus inferior designs, as represented Example Pareto Plot
( XIII-A 2001)
plots, for example, two competing MOMs (say by region C. While the Pareto plot cannot iden-
MOM 1 and MOM 2) for a point design with tify a single optimal solution, it reveals equally
one MOM each on the abscissa and ordinate. efficient designs that can be concentrated on for
The decision maker can continue to plot the re- a final series of tradeoffs
maining, competing point designs on the Pareto
plot. A useful method for doing this is to scale Rational Decision Making and Groups
the values between a good and marginal value The study of rational decision making (RDM)
where the ideal is achieved at point (1,1) and and group dynamics is outside the field of
least ideal at (0,0). Implementing this method engineering; however, it plays a vital role in all
will result in a plot similar to Figure 2. engineering decisions. Therefore, it is important
that factors influencing such decisions be identi-
By populating a plot such as this, the decision fied and considered. Two Nobel Prize winning
maker can clearly begin to see a Pareto frontier researchers in the area of RDM are Kahneman
(the curved, dashed line) emerge if enough point and Tversky. Through decades of research, they
designs are plotted. The points may be considered have repeatedly demonstrated cases in everyday
Pareto optimal if, by moving away from the point, life where people do not behave logically and
one MOM cannot be improved without degrading that these departures from rational logic occur in
the value of the second MOM. It is also important systematic patterns.
to note that the Pareto frontier is not necessarily
linear, convex, or of any specific form. Their research has identified “psychological
principles that govern the perception of decision
The Pareto frontier represents — non inferior or problems and the evaluation of options...[leading
non-dominated solutions to the MOM 1 versus to situations] in which people systematically
MOM 2 problem. These solutions are “the con- violate the requirements of consistency and
ceptual equivalents, in multiobjective problems, coherence” (Tversky and Kahneman 1981)
of a technically efficient solution in a single
objective problem” (deNeufville 1990), and are Therefore, every effort should be made to make
represented in Figure 2 by regions A and B (rep- the MCDM methodology as independent of
resenting the extreme Pareto optimums), and D subjectivity as possible; however, this cannot be
(representing the compromise Pareto optimums). avoided when examining the top-level MOMs.
All point designs that do not fall on the frontier Therefore, when performing trades of top-level
are considered dominated by those on the fron- MOMs, new methods must be used to visual-

Naval Engineers Journal SUMMER 2005 49

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

ize and perform these tradeoffs. Space does not In conclusion, this discussion of multi-criteria de-
permit a more detailed discussion of RDM and cision making illustrates a few key characteristics
group dynamics in this paper; however, more that contribute to a successful MCDM process:
information is available in (Hootman 2003). 1. The MCDM and MOM hierarchies should be
identical.
Uncertainty Considerations
2. Subjective judgments should be minimized
The probabilistic approach to MOMs advo-
and involve extensive dialogue between the tech-
cated by Crary and Green among others raises
nologists and decision makers.
an important factor that must be considered in
concept design: uncertainty. Uncertainty plays 3. Weighting schemes should be avoided when
a role in the development of the MOM metrics, used with top-level MOMs. However, weighting
on decision weights and probabilities, and in methods for rolling-up lower level MOMs can
MCDM and RDM. Thus, Zanini notes in the be used.
case of decision weights, it is important to em- 4. Uncertainty analysis should be performed.
phasize that “given the subjective and abstract
nature of [decision] weights, there is no attempt TRADEOFF METHODOLOGY
to seek a definitively right set of weights, but The concept design process of a naval combat-
rather to explore how different assumptions and ant has traditionally been accomplished using
weightings affect the relative ranking of options” rules of thumb, heuristics, accumulated expe-
(Zanini 2002). This applies to MOMs as well as rience, and parametric data, thus making it
probabilities used in the analysis as well difficult to find an ‘optimal’ solution. Further,
because of the rapid increases in technological
As Ito notes, one common method to introduce options available for ships, and a steady trend
uncertainty into the analysis is to use Monte of shrinking defense budgets over the past few
Carlo simulations: “In reality, input variables... decades, the complexity and difficulty of design
include uncertainty in nature, which could be optimization has been ever increasing.
represented by a certain probability density
function. To assess the effect of stochastic events, The identical situation has also occurred in the
Monte Carlo simulation can be used” (Ito 1995). aerospace industry over the past few decades,
A Monte Carlo simulation randomly selects and sophisticated optimization methods have
values for selected variables that have been given been developed to meet this challenge in air-
a probability distribution over a specified range. craft design. The aerospace industry has been
The Monte Carlo model then typically performs developing an increasingly popular method for
1,000 to 10,000 simulations with values chosen concept exploration using the response sur-
at a frequency consistent with the probability face method (RSM) technique. This statistical
distribution. technique identifies the design variables that
have the greatest impact on the design, and with
Rains also noted that a “discrete analysis [versus appropriate software, lead to easily manipulable
the continuous results of a probabilistic analysis] equations which can be used to define the design
would probably require a Monte Carlo technique space, conduct tradeoff studies, and facilitate
to perform the needed calculations”. (Rains 1994) better informed decision making. The example
Prior to the advent of high power desktop comput- presented will demonstrate that the application
ers, Monte Carlo simulations were very resource of a statistical method such as this RSM is effi-
intensive; therefore in earlier work, Rains used cient, cost effective, and not overly complicated.
probabilistic analyses to avoid the high computa-
tion needs of Monte Carlo. Now, the power of A parametric mapping will be done with the use
desktop computers can easily handle Monte Carlo of response surface equations (RSE) based on
simulations with commercial software packages, as regression analysis using the results from MOM
will be discussed later in this work. models. With the use of RSM, the designer will

50 SUMMER 2005 Naval Engineers Journal

develop an n-dimensional surface using a group
of techniques in the empirical study of relation-
ships between one or more measured responses
(the output variables) and a number of factors
(the input variables). The developed response
surface represents all the feasible balanced
designs. A conveniently designed statistical soft-
ware package called “JMP” by SAS Institute Inc.
was used to perform this analysis.

Response Surface Methods

RSMs concentrate on m number of factors that
have the greatest impact on the overall ship
design. To develop the RSEs, the values of the m
factors are linearly varied; however, at least three Figure 3: Three
Variable Design Models
values of each are generally used: a threshold (Goggins 2001)
(minimum value), goal (maximum value), and
middle (mean of threshold and goal values). Equation 1: Response Surface Equation

Next, point designs are developed to satisfy Where the b0,i,ii,iii terms represent constants of
either the Box-Behnken or Central Composite regression, e represents error, and the summa-
reduced order DOE models of the chosen design tions represent linear, quadratic, and interac-
space represented by the extreme threshold and tion terms respectively (Goggins 2001). This
goal values. Examples of the two models dis- equation defines the response surface and, if it
cussed here are provided in Figure 10, represent- is determined to have a statistically accurate fit,
ing a three variable design (m=3): represents all feasible concept designs.

The boxes in Figure 3 represent the design space At this point, JMP’s graphical interfaces can be
wherein it is believed an optimal solution lies used to assess all feasible design variants. Thus,
in. Thus, the Box-Behnken model avoids point with the addition of statistical modeling to the
designs at corners of the design space because concept exploration process, a finite number of
the designer believes that the corners do not designs can be used to examine an infinite num-
represent feasible designs. The model is then ber of variations of the factors.
populated with 13 point designs, 12 of which lie
between corner points, and the last at the center CONVENTIONAL SUBMARINE
of the design space. Conversely, the central com- DESIGN CASE STUDY
posite model places point designs at the corners Now that background information on the three
of the design space because the designer believes primary areas of investigation of this research
these represent feasible alternatives. This model has been discussed, a case study that ties their
incorporates 15 point designs: eight at the cor- application together can be developed and in-
ners, six in the middle of the sides of the design vestigated. The subject of this case study will be
space, and one at the center or the design space. a conventional (non-nuclear) submarine (SSK)
design problem.
After the designs are developed and the ap-
propriate model is populated, JMP is used to This section will begin by examining the role
develop the response surfaces. The response that mission analysis plays in requirements and
surface is essentially a multi-dimensional surface effectiveness analysis. Then, MOMs for a SSK
fit to the model by JMP. The response surface is will be developed following the MOM principles
defined by Equation 1: discussed earlier. Following this, the results of

Naval Engineers Journal SUMMER 2005 51

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

the application of these MOMs to the case study Explicit probabilistic formulas can be found to
will be presented using the MCDM principles answer these questions by borrowing from the
and the response surface tradeoff methodology field of operations research.
discussed earlier.
The remainder of this section will provide brief
One of the primary conclusions from the discus- discussions of the MOEs; however, detailed
sion of MOMs is that a system should be evalu- derivations and explanations of their respective
ated as it relates to a supersystem. To perform formulae are provided in Hootman (2003). It is
such an analysis, the system must be modeled important to stress that these are rough order of
in an operational context, creating the need for magnitude estimates based on simplified data.
operational analysis. In Hootman (2003) the Many technical factors, ranging from environ-
SSK was evaluated where its primary mission mental to design and operational, impact this
was an area denial and its secondary mission analysis and are not being considered in order to
was a strike and special operations force inser- simplify calculations.
tion mission. For this example, the SSK’s ability
to avoid detection, or datum search, will be This metric must relate the patrolling SSK to a
examined. Specifically, this will address tradeoffs platform and sensor searching for it. Since this
of submerged burst speed and endurance speed. situation has a moving searcher seeking a mov-
ing target, a perfect search, in which the target
Measures of Merit Development is stationary, should not be used. Therefore, the
With an understanding of the importance of primary tool for conducting this analysis will
MOMs and mission analysis in requirements be a random search. A random search is clearly
derivation, specific MOMs can be developed not the best way to conduct a deliberate search;
for the SSK case study. Kowalski et al. present a however, it is generally considered to be a good
simple, but very useful framework for doing so, lower bound for detection probability, and “often
called the goal-question-metric method. provides accurate answers”. (Washburn 1996)

The first step is to state the goal. The ultimate In this application of a random acoustic search,
goal of a system is to complete its mission, and the sensor performing the search will be treated
in the case of the SSK, its two missions are de- as a cookie cutter, that is, the sensor will sweep
fined above. Unfortunately, this will not suffice out a path at a given speed and for a given time
because this mission attainment must be quanti- with a width of twice the range of the sensor.
fied in some manner. To do this, the second step, The range of the sensor is considered to be a
re-pose the goal statement as a series of ques- positive detection range, so that if a target is
tions is used: What is the probability of the SSK outside the range it will not be detected, and if it
avoiding detection? comes within that range, it will be detected. For
the purposes of this study, a positive detection
This question can be answered by following step swath (PDS) term is created, which is a weighted
three: identify suitable measures to identify the average of snorkel and air independent propul-
extent to which each question is answered. The sion (AIP) operation detection distances based
metrics developed to answer this question are: on a submarine’s indiscretion rate.

1. STS-EB — Survivability of Suspected Target STS-EB — Survivability of Suspected

Search at the End of Burst Target Search at the End of Burst
This metric models a SSK fleeing a datum. It is
2. STS-ES — Survivability of Suspected Target based upon the assumption that the SSK has been
Search at the End of Search detected by a distant searcher who has to dispatch
an air asset to conduct the search for the SSK. The

52 SUMMER 2005 Naval Engineers Journal

formula used is a random search formula; however, it has been altered to reflect the increase in search
area over time as the sub flees the datum. The formula for MOE STS-EB is shown as Equation 2:

Equation 2: Survivability of Suspected Target Search at the End of Burst

The MOPs and DPs that are used in this equation are PDS for a sonobouy (W, nm),
Speed of Searchers (V, knots), Burst Speed (VMax), Time Late (to, hours), Burst Endurance (tB,
hours). The RSM factors in this equation are VMax and tB.

This formula includes the assumption that the searcher is not at the datum at the time of detection;
therefore, the SSK has a head start on the searcher. This is called the time late. It also assumes that
the search stops at the time that the SSK ends its burst (high speed for escape situations with corre-
spondingly low endurance)

STS-ES — Survivability of Suspected Target Search at the End of Search

This metric is very similar to STS-EB, except it is based on the more realistic assumption that the
searcher searches longer than the submarine can burst. Therefore, it must include two speeds for the
submarine, the burst speed and a slower evasion speed. The formula for MOE STS-ES is a modified
version of Equation 2 as shown in Equation 3:

Equation 3: Survivability of Suspected Target Search at the End of Search

The additional MOPs and DPs that are used in this equation Search Time (t) and Evasion Endurance
Speed (VEES, knots). The RSM factors in this equation are VMax, tB, and VEES.

Now that the two MOEs have been developed they need to be related to RSM and requirements
analysis. In RSM terms, the factors are the input variables and the responses are the MOEs. Zink et
al provides some guidance with respect to the nomenclature for requirements analysis, stating that
(Zink et al. 2000):
■ Requirements are thresholds on performance...that must be satisfied.
■ Desirements are metrics that are desired to be maximized or minimized to delineate between
competing alternatives, which satisfy the requirements.

During the process of developing the MOEs, a conscious effort was made to restrict the factors cho-
sen to DPs or MOPs that would serve as natural requirements in the design process, such as AIP and
burst endurance, balance and burst speeds, as well as weapon mix. Therefore, to facilitate a RSM
tradeoff analysis, these requirements were given a range of factor values from a threshold to a goal
value. Correspondingly, the MOEs (RSM responses) are clearly seen as desirements to be maximized.

Naval Engineers Journal SUMMER 2005 53

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

variants from combinations of the goal, thresh-

Table 2: Factors and Responses for RSM Analysis
old, and middle values of the factors.
Requirements Desirements
Factors Responses
MOE MOP/DP Threshold Goal MOEs The resulting factor matrix was copied out of
Burst Speed Survivability of Suspected JMP and inserted into a spreadsheet that applied
“Vmax” (knots) 15 25 Target Search – End of Burst the top-level MOM formulas. The resulting
Survivability “STS-EB”
of Suspected MOM values were then copied from the spread-
STS Evasion Survivability of Suspected
Target Search
Target Search – End of Search sheet and inserted into JMP to represent the
Endurance Speed 1 4
“VEES” (knots) “STS-ES” response values. After the factor and response
Time at Burst matrices have been created, JMP can perform
Speed “Tburst” (hrs) 0.5 2 multi-dimensional regressions on the data. The
results of these regressions are the response sur-
Table 2 summarizes these characteristics of this face equations for each top-level MOM.
analysis.
JMP has a number of response surface explo-
Lastly, these MOEs will be considered top-level ration and visualization tools. The one that
MOEs because the rolling up of these MOEs will captures the broadest picture is called the surface
obscure valuable insight during the tradeoff visu- plot, which displays a three-dimensional plot
alization in the next section. Further, this limited of a MOM as a function of two variables. An
example is intended to simply demonstrate the example of these is included as Figure 4.
tradeoff methodology; therefore the complexity
of a hierarchy of MOEs is not necessary. These plots illustrate the response surfaces of
the MOMs as a function of VEES and Vmax. These
Implementation of Effectiveness Analysis plots will change if any of the other factors are
Having defined the number of factors, their varied; these and other plots can be used to
range, and the responses of interest in the previ- study the design space.
ous section, the effectiveness analysis can be
implemented. The first step in this process is to However, before studying the design space, the
develop a factor matrix. JMP will perform this response surfaces must be found to be statisti-
automatically, which saves a great deal of time cally accurate. JMP performs a number of tests
with an eight-factor analysis. to determine this, but the three best indicators
are the R squared, mean, and F ratio values of
After inputting the factor ranges into JMP a cen- the regression. The first two of these are found
tral composite design was chosen because its use by using the Actual by Predicted plot of the re-
of corner points allows the best coverage of the sponse surface analysis. An example of this plot
factor ranges of interest. JMP then created the for STS-ES is provided as Figure 5.
Figure 4:
Examples of Indiviual
Response Surfaces

54 SUMMER 2005 Naval Engineers Journal

This plot shows how well the regression fits the
data supplied in the response matrix. The R
squared test represents “the proportion of the
variation in the response that can be attributed
to terms in the model rather than to random
error”. (JMP 2002) In this case, the R squared
value is 1.00, which indicates a very accurate fit.
Even though it is difficult to tell, all 145 vari-
ants are accounted for in the figure. From the
JMP output available for this example, it is clear
that the response surface provides statistically
acceptable fits. Now that the response surfaces
have been created and verified as statistically ac-
curate, the design space can be explored to show
potential design tradeoffs.

Design Space Analysis prediction trace, corresponding changes in fac- Figure 5: Actual by
Predicted Plot for STS-ES
Comprehending the visualization of a complete tors and responses can be seen.
design space in JMP is not difficult, but it can be
better understood by first examining the many The capability to manipulate the factors in this
responses that it represents individually. JMP cre- manner can illustrate the relationships between
ates a prediction profiler. that isolates the impact each variable to allow a better understanding
of every factor for every response as shown in of what factors truly drive the responses. For
Figure 6. instance, the inverse role that burst endurance
plays in STS-EB and STS-ES is not intuitively
This interactive plot is not JMP’s most elegant clear at first glance, but it is accurate. The ran-
method of presenting information, but it is one dom search equation is exponential in character,
of the most informative ones. The prediction and the datum search version of it includes an
profiler displays prediction traces (predicted area factor that increases with time.
responses as one factor is changed while holding
the others constant, represented by the black The inverse relationship of STS-EB and burst
lines in each box) for each factor along the ab- endurance is a factor of the time-late (delay
scissa. As a factor is changed, JMP recalculates between the detection of the SSK by a surface
the prediction traces to show the impact of the ship, and the arrival of an air asset to locate the Figure 6: JMP
Prediction Profiler for
change on the responses along the ordinate. SSK) and is partly an artifact of the simplicity of Top-Level MOMs

A flat line, near zero slope, indicates that a par-

ticular factor has no interaction with a certain
response. This is expected in many of these cases
because of the way the MOM formulas were
created, for instance, VEES is not a factor in
the STS-EB formula, therefore it is logical that
this factor would not impact the response. The
threshold and goal values of each factor are
reflected as the extreme values in each box, and
the current value is displayed between them. The
same applies for the responses on the ordinate,
except their extremes have been calculated by
JMP. By moving the hashed crosshair along any

Naval Engineers Journal SUMMER 2005 55

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

Figure 7: Contour
Plots of STS-EB (left)
and STS-ES (middle),
and Both (right)

the analysis. For an example, take the extreme For instance, from the prediction profiler it is
case of the burst lasting for the duration of the clear that evasion endurance speed does not have
time-late. Following the assumption that the a major impact on either STS-EB or STS-ES.
search ends when the burst ends, the searcher Therefore, tradeoffs between these two respons-
never even gets to start looking for the SSK. es should focus on burst endurance and speed.
This is not realistic, but explains the behavior
of the equation. As burst endurance increases, With the aid of contour plots, contours of values
the search rate of the searcher overtakes the area of each MOM can be seen in relation to their
created by the time-late decreasing the impact of factors, similar to a topographic map. These
burst endurance. contours represent feasible and infeasible regions
with respect to the two variables, as well as in-
A more realistic example is seen by the reverse cremental contours of STS-EB and STS-ES. The
trend in STS-ES because, following the burst, the feasible side is the side of each solid line with the
SSK is operating at its slow, evasion endurance dots. To gain further insight, these contours can
speed, adding much less area to the search as be plotted simultaneously as shown in Figure 7.
time goes by. Since the total search time is con-
stant, burst endurance determines the amount To better represent an analysis where require-
of time that the searcher (who’s search rate is ments are being discussed, regions of the contour
constant) has to search while the SSK is at the plot can be excluded from the design space by
much slower speed. This increases the probabil- setting low and high limits of acceptability for
ity that the searcher has of detecting the target. the responses. For instance, if the threshold
Therefore, in the STS-ES case, survivability is value of STS-ES is 0.6 and its goal is 0.8, and
driven by the burst speed and endurance. STS-EB’s threshold is 0.7 and goal is 0.8, the
resulting contour plots are shown in Figure 8.
This creates an interesting case of competing Lastly, the feasible design space in each of these
demands that requires a compromise solution is the white region that is not shaded. If these
two requirements were imposed simultaneously,
Considering these two examples, it is clear that the plots could be laid on top of each other as
the prediction profiler is a very powerful tool shown in Figure 8.
for an analyst or designer, but may provide too
cluttered of a picture for use by decision makers. The contour plot now shows the feasible region
Fortunately, JMP has another graphical interface that is a compromise of these two competing
that presents the actual response surfaces and is MOMS.
very suitable for use in tradeoff discussions with
decision makers. Visualization such as this is possible because of
the multi-dimensionality of RSM, which JMP
The contour plot is a visualization tool in JMP captures. As mentioned earlier, Figure 4 de-
that can simultaneously show the response picted the response surfaces of both MOMs as
surfaces with respect to two competing factors. a function of VEES and Vmax. This is one of the

56 SUMMER 2005 Naval Engineers Journal

primary tradeoffs that should be considered in
SSK design; therefore, some contour plots will be
produced to discover some relationships.

Now that the design space has been identified,

an interactive decision making process can begin.
Groups of decision makers can explore the
boundaries and interiors of design spaces with the The Monte Carlo simulation performed in this Figure 8: Threshold
ease of moving a cursor and a few slider bars in example used the probability distributions on and Goal Limits on
the JMP interface to continue to create contour the three input factors as shown in Table 3. Contour Plots (left and
middle), Compromise
plots to perform tradeoffs. In the process of doing
Design Space (right)
this, decision makers can begin to understand the The results of most interest from an analysis
constraints that mutually conflicting attributes such as this are the reverse cumulative charts,
place on the military effectiveness of the system. which show the probability distribution of
Thus, an evaluation of technologically grounded forecasted MOM values based on the predicted
alternatives is easily integrated into a require- probability distributions placed upon their
ments analysis to create a requirements space. respective factors.

Application of Uncertainty Analysis For instance, based upon the assumed distribu-
This section will present an example of Monte tions of burst speed, burst endurance, and eva-
Carlo simulation and response surface equations sion endurance speed, STS-ES has a 100% prob-
integrated to model uncertainty, a key factor in ability of a MOM value of 0.48 and a virtually
decision making. 0% probability of achieving a MOM value of
0.83. The reverse cumulative shows the middle
The response surface equations were modeled ground between these extremes in Figure 9.
in a spreadsheet and a Monte Carlo simula-
tion was performed. Probability distributions For instance, if the threshold value of STS-ES
were chosen to model each factor, including the is 0.6 and its goal is 0.8 as in Figure 8, then the
distribution shape, extreme values, and most probabilities of achieving the threshold value
likely value. Then, simulations were performed is 85% and the goal value is 7%. While the
(5000 in this case) with values randomly addition of uncertainty analyses may make the
selected at a frequency that will simulate the consideration of multiple criteria initially more
probability distribution well. difficult, it allows the decision makers to make a
more informed decision.
While the program is doing this, the response
surface equations simultaneously calculate their Unified Tradeoff Environment
values based on the randomly picked factors, The case study discussed above illustrated a
and the resulting responses are compiled. Once versatile, decision maker-friendly methodology
all of the simulations have been run, the model for exploring the impact of design requirements
reports frequency distributions, cumulative on the effectiveness of a SSK. With that case
plots, reverse cumulative plots, and statistical study in mind, a sophisticated framework for the
information on each of the responses. implementation of an expanded version of the
analysis will be discussed.

Table 3: Monte Carlo Factor Distribution Information

Factor Threshold Goal Likeliest Distribution
Burst Speed “Vmax” (knots) 15 25 20 Triangle
STS Evasion Endurance Speed “VEES” (knots) 1 4 2.5 Triangle
Time at Burst Speed “Tburst” (hrs) 0.5 2 1 Triangle

Naval Engineers Journal SUMMER 2005 57

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

to visualize the UTE is to place three prediction

profilers side by side, as in Figure 10.

Examination of the design problem in this man-

ner allows the simultaneous consideration of the
effects of each of the three factor sets on system
constraints and objective responses.

Mavris and DeLaurentis provide an overview of

how the UTE is developed. First, a baseline set
of each of the factors is determined. Then, the
Prior to describing an improved framework for requirements space is developed with the design
Figure 9: Reverse
Cumulative Distribu- the supersystem, the system must be revisited. As variables and K-factors held constant at their
tion of STS-ES mentioned earlier, this analysis did not involve baseline. Likewise, when the design variable
an engineering model to validate the variants space is developed, requirements and K-factors
that were developed. Further, the methodology are held at baseline, and a similar method is
did not integrate any consideration of the impact used when developing the K-factor space. This
of future advances in technological capability, results in three sets of response surface equations
such as improved propulsion systems. that can be “aggregated into an overall expres-
sion for changes in desirements as a function of
These two oversights were intentional for this requirements, design/economic variables, and
analysis, but are essential for achieving a balanced technology improvements” (Mavris and DeLau-
understanding of, and design for, the system un- rentis 2000). Another example of the flexibility
der consideration. To do so, the response surface and application of the UTE equations is demon-
methodology must incorporate three groups of strated by the following statement:
factors: concept design variables, requirements,
and technology K-factors. The first two are equation sets can be interchanged and sub-
intuitively clear, but the technology K-factor is sequently fed to a non-linear, simultaneous
less clear. This K-factor is a factor that is inserted equation solver to determine if solutions exist in
into the engineering model to represent a pre- the aspiration space....For example, one could
dicted notional degradation or improvement to fix the requirements and conduct a search over
various technologies based on future research and evolutionary technologies and design variables
development. By introducing these factors, the to achieve the goals. Alternatively, the design
analysis integrates the impact of future advances variables can be fixed while the search is over
in technological capability. the requirements and technology levels. (Mavris
and DeLaurentis 2000)
The simultaneous combination of the design
variables, requirements, and K-factors creates The characterization of the design, requirements,
Figure 10: The what Georgia Institute of Technology’s Aero- and technology spaces into a unified tradeoff
Unified Tradeoff Envi- space Systems Design Lab terms the unified environment introduces a much more rigorous
ronment (Soban and
Mavris 2000a) tradeoff environment (UTE). A convenient way analysis into the traditional design process.

Further, the UTE can play an important role in

the process of requirements tradeoff and defini-
tion “where the requirements study can be used
to determine which specific point in a require-
ments space the system is to fall. This can be
performed using integrated product and process
development” methods (Hollingsworth and Ma-

58 SUMMER 2005 Naval Engineers Journal

vris 2000) grounded in a sound group decision
making strategy

Expanded Effectiveness Analysis

The creation of the UTE will play a key role in
the development of an expanded effectiveness
analysis framework because it brings more infor-
mation to the analysis process. The effectiveness
models developed for this study are extremely
crude and elementary ones, focusing primarily
on the single platform under consideration, but
did make the necessary steps to fully place the
SSK into an operational context. As discussed in Figure 11: System of
the section describing MOM principles, this is a naval ship design and acquisition by wrapping Systems Approach (So-
key factor for a proper effectiveness analysis. effectiveness analysis, multi-criteria decision ban and Mavris 2000a)
making, and response surface methods into one
This approach requires the analysis of factors part of an advocated effectiveness framework
internal and external to system boundaries, creat- and methodology.
ing a system of systems approach that “is based on
existing probabilistic methodologies that define the The need for such an analysis firmly grounded in
[system]...[and the] extrapolation of these methods the principles of systems engineering and require-
to the theater level...redefining the system as the to- ments was established. The unified tradeoff en-
tal warfighting environment”. (Soban and Mavris vironment framework and effectiveness tradeoff
2000a) Thus, a virtual response surface hierarchy methodology advocated by this research facili-
can be created as shown in Figure 11. tates an informed negotiation of requirements,
desirements, and design parameters by decision
By using the probabilistic system of systems ap- makers. This process allows vehicle design and
proach grounded on solid MOM and MCDM mission requirements, “when optimized to maxi-
principles, better systems can be designed. Instead mize the overall effectiveness of the system, [to]
of designing the system “to its own pre-defined become the requirements to which the vehicles are
performance and mission constraints, [it] can then designed” (Soban and Mavris 2000a). This
now be optimized to fulfill theater level goals and can be further generalized to the whole concept
objectives”. (Soban and Mavris 2000a) design framework to show that the objective is
not to develop a single absolute optimum, rather
To further improve the MCDM philosopy, fur- it is elicit relationships for determining what char-
ther research into meta model methods of inter- acteristics have the greatest impact on the design,
action and negotiation for design, effectiveness, why they do, and how these relationship can be
and requirements tradeoffs should be explored. better exploited to lead to a better design.
This should be conducted as an investigation to
determine the state of the art of such methods in This represents a profound improvement over
both naval and non-naval industries and orga- traditional, ad hoc tradeoff methodologies,
nizations. Application of more mature methods which rely on a limited number of point designs
to allow real-time what if excursions will further and data. The design space meta model pro-
facilitate informed decision processes and more vides a continuous, interactive design space
effective designs. examination tool that can be used in real time
by decision makers to explore and negotiate the
Conclusions “simultaneous impact of requirements, product
This research has set the stage for performing design variables, and emerging technologies dur-
a military effectiveness tradeoff analysis for ing the concept formulation and development

Naval Engineers Journal SUMMER 2005 59

A Military Effectiveness Analysis and Decision Making Framework for Naval Ship Design and Acquisition

stages”(Zink et al. 2000) to reach compromise Hollingsworth, P., Mavris, D.N., “A Method for Concept Ex-
design solutions. ploration of Hypersonic Vehicles in the Presence of Open
& Evolving Requirements,” 5th World Aviation Congress
and Exposition, San Diego, CA, 2000.
Acknowledgements
The authors would like to thank Ms. Katherine Hootman, John C., “A Military Effectiveness Analysis and
Drew of the Office of Naval Research (Code Decision Making Framework for Naval Ship Design and
334, Ship Systems and Structure Division) for Acquisition,” MIT Thesis, June 2003.
the financial support of this research, as well Ito, Hideto, “A Study of the Measures of Effectiveness
as Dr. Hank Marcus and Konstantinos Psal- for the JMSDF AEGIS Destroyer In a Littoral, Air Defense
lidas for their contributions when this research Environment,” Naval Postgraduate School Thesis, 1995.
was performed at the Massachusetts Institute “JMP:The Statistical Discovery Software,” 2002 SAS
of Technology. The views expressed herein are Institute Inc.
those of the authors, and not necessarily those of
Keeney, R., and Raiffa, H., “Decisions with Multiple Objec-
the U.S. Navy or Department of Defense. ■ tives,” Wiley, New York, 1976.
Leite, M. J., and D. R. Mensh, “Definition of Evaluation
REFERENCES
Criteria for System Development Acquisition Modeling
Brown, Kevin W., “Measuring the Effectiveness of
and Simulation,” Naval Engineers Journal, January 1999.
Weapons Systems in Terms of System Attributes,” Naval
[Leite and Mensh, 1999]
Postgraduate School Thesis, 1995.
Leopold, Reuven, “U.S. Naval Ship Design: Platforms vs.
de Neufville, Richard, “Applied Systems Dynamics,”
Payloads,” U.S. Naval Institute Proceedings, August 1975
McGraw-Hill, 1990.
Mavris, D. M., and D. DeLaurentis, “Methodology for
Don, Bruce W., Thomas Herbert, and Jerry Sollinger,
Examining the Simultaneous Impact of Requirements,
“Future Ground Commanders’ Close Support Needs
Vehicle Characteristics, and Technologies on Military Air-
and Desirable System Characteristics,” RAND, MR-833-
craft Design,” 22nd Congress of the International Council
OSD, 2002
on the Aeronautical Sciences (ICAS), Harrogate, England,
Frits, A., N. Weston, C. Pouchet, A. Kusmik, W. Krol, Jr., and August 27-31, 2000.
D. N. Mavris, “Examination of a Torpedo Performance
Mine Warfare Center, “MCM Measures of Effectiveness
Space and its Relation to the System Design Space,”
(MOE’S) and Measures of Performance (MOP’S),” Unclas-
American Institute of Aeronautics, 2002.
sified, A-2G-2758.
General Accounting Office, “Best Practices: Better Match-
MIT Naval Construction and Engineering Program (XIII-
ing of Needs and Resources Will Lead to Better Weapon
A), “Lecture #2: Effectiveness Estimating,” Principles of
System Outcomes (GAO-01-288),” United States General
Naval Ship Design Class Notes, 2001]
Accounting Office, March 2001.
OAS - Office of Aerospace Studies, “AoA Handbook: A
Goggins, David A. “Response Surface Methods Applied
Guide for Performing an Analysis of Alternatives (AoA),”
to Submarine Concept Exploration,” MIT Thesis, 2001
Office of Aerospace Studies, Air Force Materiel Command
Green, John M., Establishing System Measures of Ef- (AFMC) OAS/DR, June 2000.
fectiveness,” Proceedings of the AIAA Biennial National
Rains, Dr. Dean A., “Combatant Ship Design Guidance
Forum on Weapon System Effectiveness, March 2001(a)
Through Mission Effectiveness Analysis,” Naval Engi-
Green, John M., “Modeling the Ship as a Weapon Sys- neers Journal, May 1984.
tem,” 69th MORS Symposium, Annapolis, Maryland, June
Rains, Dr. Dean A., “Methods for Ship Military Effective-
2001 (b).
ness Analysis,” Naval Engineers Journal, March 1994
Green, John P. and Bonnie W. Johnson, “Towards a The-
Rains, Dr. Dean A., “Fleet Mix Mission Effectiveness
ory of Measures of Effectiveness,” 2002 Command and
Analysis,” Naval Engineers Journal, January 1999
Control Research and Technology Symposium, Monterey,
Calif., June 11-13, 2002. Saaty, T., “The Analytical Hierarchy Process,” University
of Pittsburgh, 1988
Hockberger, William A., “Total System Ship Design in a Su-
persystem Framework,” Naval Engineers Journal, May 1996. “Soban, Danielle S., and Dimitri N. Mavris, “Formulation

60 SUMMER 2005 Naval Engineers Journal

of a Methodology for the Probabilistic Assessment of Whitcomb, Cliff, “A Prescriptive Production-Distribution
System Effectiveness,” Aerospace Systems Design Labora- Approach for Decision Making in Product Design Engi-
tory, Georgia Institute of Technology, 2000 neering,” University of Maryland Ph.D. Thesis, 1998 (b).
Tversky, Amos and Daniel Kahneman, “The Framing of Zink, P.S., Mavris, D.N., Raveh, D.E., “Integrated Struc-
Decisions and the Psychology of Choice,” Science, Vol. tural/Trim Optimization for Active Aeroelastic Wing
211, Issue 4481, 453-458, 1981 Technology,” 8th AIAA/NASA/USAF/ISSMO Symposium
on Multidisciplinary Analysis and Optimization, Long
Washburn, Alan R., “Search and Detection, 3rd Ed.,”
Beach, Calif., 2000
Institute for Operations Research and Management Sci-
ences, 1996. Zanini, Michele, “Italy’s All-Volunteer Army: An Analytical
Framework for Understanding the Key Policy Issues and
Whitcomb, Cliff, LCDR, USN, “Naval Ship Design Philoso-
Choices During the Transition,” RAND, RGSD-162, 2002.
phy Implementation, Naval Engineers Journal, January
1998 (a)

John C. Hootman is a naval architect for future ship concepts at the Naval Surface Warfare Center’s
Carderock Division. He holds a bachelor of science in naval architecture and marine engineering from Webb
Institute (2001) and master of science in naval architecture and marine engineering, as well as ocean systems
management from the Massachusetts Institute of Technology (2003). At Webb Institute, John received the
Award for Excellence in Engineering Design, Lewis Nixon Memorial Prize (best naval architecture thesis),
C. A. Ward Jr. Memorial Award (highest naval architecture average), and the R. A. Partanen Humanities
Award. John was a MIT Presidential Fellow for the 2001/2002 academic year, and his graduate research
focused on concept design trade-off and decision making methodologies with an emphasis on military ef-
fectiveness analysis and requirements determination.
Since joining Carderock, John has supported a SEA 017 independent cost analysis for the Littoral Combat
Ship Program as well as a NSWC Dahlgren Division Sea Basing force architecture analysis. He has also
spent four months at sea attached to Carrier Strike Group FIVE’s Flag Operations Staff aboard the USS
Kitty Hawk in the Western Pacific and has worked with NAVAIR’s Aviation/Ship Integration Department.
He is currently supporting PEO Integrated Warfare Systems’ Chief Warfare Systems Engineer for Amphibi-
ous, Auxiliary, and Coast Guard Ships.
John has previously served as a member of SNAME’s Education Committee and as the vice chair of SNAME’s
Student Steering Committee and is a member of ASNE and the U.S. Naval Institute. He has also received the
2002 SNAME Undergraduate Honor Paper Prize and 2003 SNAME Graduate Honor Paper Prize.

Cliff Whitcomb Until recently Cliff Whitcomb has been a professor of naval architecture and marine
engineering, and the Northrop Grumman Ship Systems Endowed Chair in Shipbuilding and Engineering in
the School of Naval Architecture and Marine Engineering, and professor of engineering management, at the
University of New Orleans, New Orleans, Louisiana. In addition, Dr. Whitcomb is an eminent scientist in
naval electric systems at the Naval Surface Warfare Center, Carderock Division (NSWCCD) Philadelphia
Detachment.
Prior to his position at UNO, he was an associate professor of naval construction and engineering for the
MIT 13A program, and a senior lecturer and research scientist in the System Design and Management
(SDM) program in the Engineering Systems Division (ESD) at MIT. Dr. Whitcomb taught and performed
research in the areas of product design and development, ship design, systems engineering, multidisciplinary
design optimization, and multiple criteria decision making as they apply to product design and develop-
ment. He is a Six Sigma Master Black Belt for Northrop Grumman Ship Systems and a certified systems
engineering professional.
Dr. Whitcomb has a Ph.D. in mechanical engineering from the University of Maryland in 1998, a naval
engineers degree and a masters degree in electrical engineering and computer science from MIT in 1992, and
a bachelors degree in engineering from the University of Washington in 1984.
Dr. Whitcomb has recently accepted a professorship at the Naval Post Graduate School.

Naval Engineers Journal SUMMER 2005 61