COST-EFFECTIVENESS

ANALYSIS
 COST-EFFECTIVENESS
ANALYSIS
A Systems Engineering Perspective

Howard Eisner
First edition published 2022
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
© 2022 Howard Eisner
CRC Press is an imprint of Taylor & Francis Group, LLC
Reasonable efforts have been made to publish reliable data and
information, but the author and publisher cannot assume responsibility
for the validity of all materials or the consequences of their use.
The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been
obtained. If any copyright material has not been acknowledged please
write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be
reprinted, reproduced, transmitted, or utilized in any form by any electronic,
mechanical, or other means, now known or hereafter invented, including
photocopying, microfilming, and recording, or in any information storage or
retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this
work, access www.copyright.com or contact the Copyright Clearance
Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. For works that are not available on CCC please contact
[email protected]
Trademark notice: Product or corporate names may be trademarks or
registered trademarks and are used only for identification and
explanation without intent to infringe.
Library of Congress Cataloguing-in-Publication Data
Names: Eisner, Howard, 1935- author.
Title: Cost-effectiveness analysis : a systems engineering perspective /
Howard Eisner.
Description: Boca Raton, FL : CRC Press, 2022. | Includes
bibliographical references and index.
Identifiers: LCCN 2021039827 (print) | LCCN 2021039828 (ebook) |
ISBN 9781032051345 (hbk) | ISBN 9781032051352 (pbk) | ISBN
9781003196174 (ebk)
Subjects: LCSH: Cost effectiveness.
Classification: LCC TA177.7 .E46 2022 (print) | LCC TA177.7
(ebook) | DDC 658.15/52‐‐dc23/eng/20211022
LC record available at https://lccn.loc.gov/2021039827
LC ebook record available at https://lccn.loc.gov/2021039828
ISBN: 978-1-032-05134-5 (hbk)
ISBN: 978-1-032-05135-2 (pbk)
ISBN: 978-1-003-19617-4 (ebk)

DOI: 10.1201/9781003196174

Typeset in Times
by MPS Limited, Dehradun
This book is dedicated to:

• The RAND Corporation that made seminal contributions to

the fine art and science of cost-effectiveness analysis
• Eberhardt Rechtin, whose heuristics were cost-effective
suggestions
• My wife, June Linowitz, who continually makes cost-
effective choices
• My children, Oren and Susan, who lead cost-effective lives
• My five grandchildren, Jake, Gabe, Rebecca, Zachary, and
Ben, all in their 20s, who have all chosen cost-effective paths
Contents

Preface ix
Author xi
Books by the Author xiii

1 WSEIAC, the Whiz Kids, and RAND 1

2 An Overview of the Basics 13

3 Cost-Effectiveness and Systems Engineering 23

4 Selected Commentaries on Cost-Effectiveness 31

5 Life Cycle Costing 43

6 Cost-Estimating Relationships 51

7 Cost-Effectiveness and Benefits 59

8 Everyday Cost-Effectiveness 67

9 Selected Military Examples 73

10 Miscellany 83

11 Summary 93

Index 101

vii
Preface

Two key aspects of systems engineering are “synthesis” and “analysis.” The
synthesis part is largely, and correctly, associated with systems architecting. The
analysis part can be thought of as all the systems engineer does in order to verify
that the system will satisfy the requirements. Four references that help us do
analysis are those by Blanchard and Fabrycky, Scherer, Buede and the SERC’s
SEBoK. Essentially missing from much on the subject of systems engineering
analysis is that of “cost-effectiveness.” This book is intended to give the systems
engineer a preliminary source in that regard. It also indicates when and how the
systems engineer engages in cost-effectiveness analysis.
In addition, the topic of cost-effectiveness has both deep and broad applicability.
It is a topic that can be used with relative ease to support decisions made at work, at
home, and just about wherever one can contemplate. The keys to its versatility are
its visual attraction, its format of comparison, and its simplicity of concept. Here
are some questions that can be readily addressed by using cost-effectiveness
analysis:

• What is the best system to build for my customer?

• How do I best integrate systems that are on the market and off-the-
shelf?
• What are the best machines that I can use to replace office
machines that have become obsolete (e.g., copiers, etc.)?
• What telephone system should I install in my offices?

At home:

• What automobiles should I buy for each member of my family?

• What washer/dryer combination should I install in my laundry room?
• How should I overhaul my kitchen?
• What home computer should I have?

Now that’s an important array of decisions that carry with them

considerable costs and lots of “stakeholders” to satisfy.

ix
x Preface

So the basic idea is that – in each case and situation – I look at

several alternatives, what they cost, and how effective they are likely
to be. All of this information fits conveniently on a spreadsheet with
room for footnotes and bold print, supported by a graph of cost
numbers and effectiveness measures.

So go to it, and pick the simplest form first. You won’t be disappointed.
Eberhardt Rechtin, the Master Systems Engineer was right … Keep It
Simple, Stu...d!

Howard Eisner
Bethesda, Maryland
Author

Howard Eisner spent 30 years in industry and 24 years in academia. In the former,
he was a Research Engineer, Manager, Executive (at ORI, Inc. and the Atlantic
Research Corporation), and President of two high-tech companies (Intercon
Systems and the Atlantic Research Services Company). In academia, he was
Professor of Engineering Management and a Distinguished Research Professor
in the engineering school of The George Washington University (GWU). At
GWU, he taught courses in systems engineering, technical enterprises, project
management, modulation and noise, and information theory.
He has written 12 books that relate to engineering, systems, and management.
He has also given many lectures and tutorials to professional societies (such as
INCOSE – International Council on Systems Engineering), government agencies
(such as the DoD, NASA, and DOT), and the Osher Lifelong Learning Institute
(OLLI). In 1994, he was given the outstanding achievement award from the
GWU Engineering Alumni.
Dr. Eisner is a Life Fellow of the IEEE and a Fellow of INCOSE and the
New York Academy of Sciences. He is a member of Tau Beta Pi, Eta Kappa
Nu, Sigma Xi, Omega Rho, and various honor/research societies. He received
a bachelor’s degree (BEE) from the City College of New York (1957), an MS
in electrical engineering from Columbia University (1958), and a doctor of
science degree from The George Washington University (1966).
Since 2013, he has served as Professor Emeritus of engineering management
and a Distinguished Research Professor at The George Washington University.

xi
Books by the Author

• Computer-Aided Systems Engineering

• Reengineering Yourself and Your Company
• Managing Complex Systems: Thinking Outside the Box
• Essentials of Project and Systems Engineering Management
• Systems Engineering: Building Successful Systems
• Thinking: A Guide to Systems Engineering Problem Solving
• Systems Architecting: Methods and Examples
• Systems Engineering: Fifty Lessons Learned
• What Makes the Systems Engineer Successful?
• Problem Solving: Leaning on New Thinking Skills
• Topics in Systems

xiii
WSEIAC, the
Whiz Kids,
and RAND
1
In the summer of 2019, an economist by the name of Alain Enthoven au-
thored a paper [1] on Systems Analysis (SA) and the cost and effectiveness of
weapon systems. He noted that Charles Hitch, Chief Economist of the RAND
Corporation, was appointed by Robert McNamara, Secretary of Defense, as
Comptroller of the DoD and that 1961 approximately the year of the SA thrust
in his office. McNamara also called this thrust P, A, and E (Planning,
Assessment, and Evaluation). Its main purpose was to evaluate DoD systems
in terms of their value to the DoD, and make PPBS (Planning, Programming,
and Budgeting) decisions as to which systems are needed to be built and
which are not.
The Enthoven paper pointed out that, by 1965, three milestones occurred:

1. Hitch became the president of the University of California,

Berkeley.
2. Enthoven was appointed Assistant Secretary of Defense for
Systems Analysis (SA).
3. The DoD published their WSEIAC report [2], a definitive treatise
on how the DoD would deal with weapon systems and their ef-
fectiveness. The key summary report defined weapon system ef-
fectiveness as the product of A, D, and C, where:

A = Availability
D = Dependability
C = Capability

These important steps set the stage for the calculation of the “effectiveness”
part of “cost-effectiveness” analysis.

DOI: 10.1201/9781003196174-1 1
2 Cost-Effectiveness Analysis

At approximately that time (1965–66), this author found himself in front of

Alain Enthoven, exploring the possibility of moving from a small think-tank
company in Silver Spring, Maryland (Operations Research Inc.), to Enthoven’s
staff as part of the “Whiz Kids” in McNamara’s P, A, and E. The discussion
was quite exciting but ultimately did not lead to a change in positions into the
SA part of the DoD. I don’t recall the details of that conversation, but at this
time I would call it at least a mismatch of needs and responsibilities associated
with this author’s career and work interests.
In this reporting, this author is using “Whiz Kids” as the generic name of
all the folks that inhabited the P, A, and E portion of McNamara’s DoD
responsibilities. It was indeed a very key portion in terms of the analyses and
evaluations that were being carried out there. And the WSEIAC report played
an important role at that time in setting the stage for the effectiveness as-
sessments of military systems.

THE WSEIAC ANALYSIS AND REPORT

This report had several special parts:

• The principal findings, conclusions, and recommendations of the

five WSEIAC Task Groups
• A 15-step procedure for cost-effectiveness assurance
• A section of the integrated summary that contains abstracts and
summaries of each of the ten reports submitted by the five Task
Groups
• Appendix I, which has a more detailed treatment of each of the ten
report tasks
• Appendix II, which contains an application of the metho-
dology, and
• Appendix III, which is a glossary of cost and effectiveness terms

To continue on with the top-level definition:

Effectiveness is defined as the product of Availability, Dependability and

Capability, where
“Availability is a measure of the condition of the system, at the start of a
mission, when the mission is called for at an unknown (random) point in time,”
 1 • WSEIAC, the Whiz Kids, and RAND 3

“Dependability is a measure of the system condition during the performance

a the mission, given that it is available at the start of the mission, and”
“Capability is a measure of the results of the mission, given the condition of
the system during the mission (dependability)”
“Cost-effectiveness is the value received (effectiveness) for the resources
expended (cost)”

A concise statement of the perceived problem:

a minimum acceptable requirement of a recent SOR is given piecemeal in

terms of separate probabilities and performance limits without obvious re-
lation to one another. When combined into an overall effectiveness number
(using WSEIAC notions) these requirements suggest that if this system
works less than 4 times out of 100, it is still acceptable. [2, page 5]

The overall Committee for the WSEIAC was composed of five task groups of
about ten members each. The objectives of each Task Group are delineated as
follows:

Task Group I
1. Review present procedures for effectiveness requirements.
2. Recommend a method for determining system effectiveness re-
quirements that are mission responsive.

Task Group II
1. Review existing documentation of system effectiveness.
2. Recommend methods and procedures for measurement and pre-
diction of system effectiveness in all phases of a system’s life.

Task Group III

1. Review current Air Force data collection and reporting systems.
2. Recommend uniform procedures for periodic status reporting to
assist all management levels.

Task Group IV
1. Develop a set of basic instructions and procedures for conducting
analysis for system optimization, considering:
• Effectiveness
• Cost
• Program time scale
• Refine current cost-effectiveness analysis techniques
4 Cost-Effectiveness Analysis

Task Group V
1. Develop a management system designed to absorb and apply sys-
tems effectiveness experience retention.

TECHNICAL REPORTS
• AFSC – TR – 65 – 1: Final Report of Task Group 1 “Requirements –
Methodology”
• AFSC – TR – 65 – 2: Final Report of Task Group II: “Prediction –
Measurement”
• AFSC – TR – 65 – 3: Final Report of Task Group III: “Data
Collection and Management Reports”
• AfSC – TR – 65 - 4: Final Report of Task Group IV: “Cost-
Effectiveness Optimization”
• AFSC – TR – 65 – 5: Final Report of Task Group V: “Management
Systems”

The top-level report is provided in two volumes:

Volume I – General Summary

Volume II – Integrated Summary

So for the timeline of this author, WSEIAC plays an important role, especially
when observed from the perspective of a starting point for cost-effectiveness
analysis. After all, one might argue that there is no better place to start from
other than the treatise that defines effectiveness. So, we move forward from
there to the WSEIAC recommendations, which in effect is the last statement
of that AFSC report, back in 1965.

WSEIAC RECOMMENDATIONS
These are the essential recommendations of this significant report from the
Air Force.
 1 • WSEIAC, the Whiz Kids, and RAND 5

1. The basic analytical framework and effectiveness concepts should

be accepted as an initial starting base.
2. The proposed effectiveness notions and evaluation techniques need to
be validated on one or more Air Force systems. Changes in the
techniques should be based upon the results of that validation.
Contractual commitment needs to be, likewise, the validation results.
3. Procedures for test and demonstration of effectiveness, similar to
those for reliability, should be initiated. (Another good starting point.)
4. The matter of “confidence” in the effectiveness results need to be
further studied.
5. Specific impacts on incentive fee contracts need to be explored as
soon as possible.
6. The preparation and adoption of standard terminology for systems
effectiveness needs to be accomplished as soon as possible.
7. More attention needs to be paid to data acquisition (e.g., main-
tenance data).
8. The fine art of parameter estimation needs further definition and
standardization.

MORE ABOUT THE WHIZ KIDS

McNamara’s tenure as Secretary of Defense under Kennedy and Johnson was
notable, as he established the Whiz Kids in the DoD. Stories of this period of
time are mixed, however. On one hand, this collection of people was out-
standing, and there was a set of new ways to address military system and
force levels. This was the new era of quantitative analysis, with new respect
for and knee-deep systems analysis, econometrics, cost-effectiveness ana-
lysis, and PPBS methodologies.
The negative side of the activity, however, was acknowledged to be-
coming mired down “in the fog of war.” McNamara and his people lost sight
of the need to bring the war to a halt, responding to the outcries of the public
at large. So, what might a bottom line be for that time and the contributions
made or not made by this special secretary of defense?
An article in the Harvard Business Review summarizes by declaring that
McNamara’s time represented the “evolution of Modern Management” [3].
There was general acceptance of quantitative analysis leading to the creation
of increased shareholder value for the enterprise. But it was also true that
during that time there was a new focus on contributions to society and on
6 Cost-Effectiveness Analysis

figuring out and implementing programs for the common good. Despite
McNamara’s being the “architect” of a futile war, he was also president of the
World Bank for some 13 years. So he was able to “change his tun,” appro-
priately, during different phases of his life and position. He brought rigor to
the statistical analysis of the war effort, and then brought that same rigor to
trying to “make the world a better place.” A Business Week article used the
following phrase to characterize McNamara and his style of leadership.

he was a prize specimen of a remarkable breed in U.S. Industry – the trained

specialist in the science of business management who is also a generalist
moving easily from one technical area to another.

A Barry Goldwater quote was sharper: “McNamara was one of the best se-
cretaries ever, an IBM machine with legs.”
In the award-winning documentary, McNamara provided a short synopsis
of his approach in The Fog of War [4]: his two key principles were:
“Maximize efficiency” and “Get the Data.” Other interesting points set forth
by McNamara in that treatise were:

a. Empathize with your enemy

b. One is not able to change human nature, and
c. Use proportionality to guide your behavior in war

This author agrees with (a) and (b) mentioned earlier but not (c), for which we
use the atom bombing as a counterexample.

HOW MUCH IS ENOUGH?

A very instructive treatise on McNamara’s time as Secretary of Defense is
provided in Enthoven’s book (with Smith) How Much Is Enough [5].
McNamara was 100% devoted to bringing strong analytic methods to military
systems planning, programming, and budgeting. The Foreword to How Much
Is Enough reveals key ideas about the book in the following categories:

a. Principles
b. Draft Presidential Memorandums (DPMs)
c. The Planning, Programming, and Budgeting (PPB) system
d. Four Important Trends
e. New Processes
 1 • WSEIAC, the Whiz Kids, and RAND 7

Six principles have been identified as:

1. Decisions should be based upon explicit criteria of national interest

and not on compromises between institutional forces.
2. Needs and costs should be considered at the same time.
3. The choices need to be explicit, balanced, and feasible.
4. SecDef needs to have an active analytic staff to provide relevant
and unbiased data.
5. A multiyear plan must contain a projection of the consequences of
present decisions.
6. All analyses need to be open, explicit, and available to all.

Ninety-three DPMs were issued during the Kennedy–Johnson years, shaping

the Nuclear Triad. A 1962 DPM is the only one that has become open release,
dealing with the Strategic Retaliatory Forces.
Continuation of the PPB system remains the essential substance of DoD
Secretary of Defense management.

FOUR IMPORTANT TRENDS

Four important trends have been articulated as:

1. There has been greater congressional involvement in DoD

management.
2. There has been an increase in the services’ analytic capabilities.
3. There has been an increase in the advocacy from OSD staff, in
general.
4. There has been an increase in the involvement of the Joint Staff and
the heads of the major commands.

NEW PROCESSES
The book remains a “beacon” for DoD SecDef management, even after some
35 years. Each new SecDef appears to reserve the right to modify various
processes to his liking and ask for new investigations. For example, in 2003,
8 Cost-Effectiveness Analysis

Rumsfeld requested, and got, a Joint Defense Capability Study. Another

example is the fundamental change obtained by SecDev William Perry re-
quiring the use of Integrated Product and Process Development (IPPD), and
the use of Integrated Product Teams (IPTs).

THE WHIZ KIDS – SOME LAST WORDS

After WWII, the ten Air Force officers that ran stat control sold themselves to
Henry Ford II as he tried to modernize the company. What follows here is a
thumbnail of the backgrounds for each of the ten Whiz Kids, including
McNamara himself. It was, indeed, an outstanding group of leaders and
analysts. Alain Enthoven is also listed although he was technically not one of
the original Whiz Kids. Reason? He played a major role in the DoD as the
Assistant Secretary of Defense for Systems Analysis.

Alain Enthoven – He was the “Chief Economist” with a BA from

Stanford, an MPhil from Oxford, and a PhD from MIT who defined the sum
and substance of SA as a discipline and applied it in the Office of the
Secretary of Defense.
Tex Thorton – Tex was fired from Ford as he continually clashed with an
old-timer named Lewis Crusoe. He went on to head Litton Industries and
created the first “conglomerate,” which he ran successfully for many years.
He received the Medal of Freedom from President Reagan before he passed
away in 1981.
Jack Reith – Jack Reith was offered and accepted the position of the
President of Ford in France. He returned to headquarters and led the Mercury
Division with the Turnpike Cruiser, which turned out to be a disaster. He
never recovered from that disappointment.
Arjay Miller – Arjay Miller was offered the presidency of Ford, which he
accepted and ran successfully for some years. From there, he moved on to
become the Dean of Stanford’s Business School.
J. Edward Lundy – Mr. Lundy became Ford’s CFO and trained thousands
of Ford people (mostly MBAs) from 1967 to 1979.
Ben Mills – Mr. Mills became Vice President and General Manager of
Ford’s Lincoln Division, and retired from Ford in 1971.
James Wright – Mr. Wright rose at Ford where he became Group Vice
President of the Car and Truck Divisions
Charles Bosworth – Charles Bosworth headed Ford Purchasing Division
and retired in 1966 at the age of 51.
 1 • WSEIAC, the Whiz Kids, and RAND 9

George Moore – George Moore quit in 1947 to buy part ownership of a

Washington, DC Lincoln Mercury dealership.
Gene Andreson – Mr. Andreson left Ford within a year as he was un-
happy with the corporate environment and the cold winter.

Robert McNamara was the “best of the brightest” (of the Whiz Kids) and
became Henry Ford’s choice for President of the Ford Motor Company and
also J. F. Kennedy’s selection for Secretary of Defense. All ten were hired out
of Harvard by Henry Ford who recognized their superior skills and the needs,
at that time, of the Ford Motor Company.
It is claimed that the Whiz Kids made key contributions to the workings
of Systems Analysis in the Office of the SecDef, but also made three basic
errors [6]. Error number one had to do with failure to recognize the im-
portance of key drivers such as product quality, customer loyalty, and the
value of R&D investments. The second error area had to do with missing
other key drivers that provided the greatest insight into the warfare scenario
and human behavior. Finally, there was a general failure to project the effects
of key drivers and variables into the future.

THE RESEARCH AND DEVELOPMENT

(RAND) PROJECT AND CORPORATION
This author makes the observation that the RAND Corporation is an en-
terprise that appears to have survived by being in the right place at the right
time and with an appropriate purpose and charter. Going back to the post-
WWII period, it was the Commanding General of the Army Air Force that
made the observation in a report to the Secretary of War [7,8]:

during this war, the Army, Army Air Forces, and the Navy have made
unprecedented use of scientific and industrial resources. The conclusion is
inescapable that we have not yet established the balance necessary to insure
the continuance of teamwork among the military, other governmental
agencies, industry, and the universities. Scientific planning must be years in
advance of the actual research and development work.

This led almost immediately to the establishment of Project RAND under

special contract to the Douglas Aircraft Company, in Santa Monica, California.
By 1948, Project RAND had grown to 200 staff, a modest beginning. It was
operating successfully under a quite simple set of articles, namely to:
10 Cost-Effectiveness Analysis

further and promote scientific, educational, and charitable purposes, all for
the public welfare and security of the United States of America.

Also in 1948, Project RAND was officially transferred from the Douglas
Aircraft Company to the RAND Corporation.
It started out with a major emphasis on solving military problems, but
later moved into social programs. The military work was critically important
in that it proved the ability of the RAND to contribute in an arena that needed
a lot of help. It also proved that an enterprise with the following key attributes
had a significant place in that world:

• Nonprofit
• Nonpartisan
• Objective
• Uses best analytical tools and methods
• Produce work of the highest quality
• Interdisciplinary

Other highlights to be noted is the early work of RAND in such areas as [7]:

• Space systems
• Digital computing
• Early computers
• Packet switching
• Decision making under uncertainty
• Game theory
• Linear and dynamic programming
• Modeling and simulation
• Artificial intelligence (AI)
• Network theory
• Cost analysis
• The overall field of systems analysis

So, at this point, we find a linkage between RAND and Robert McNamara
and his Whiz Kids. Apparently, RAND developed the Planning,
Programming, and Budgeting System (PPBS). This was the very same
system that McNamara used in the early 1960s. So, we had an intermingling
of methods and ideas at that time, to everyone’s benefit and advantage. It
was a way of doing business in the Pentagon, across the board. It set forth
principles of PPBS that were crucial in moving forward with military
matters during those critical years.
 1 • WSEIAC, the Whiz Kids, and RAND 11

As we close with the “story” of RAND, we also take note of the three
areas in which RAND made super-critical contributions, according to the
literature. The first was coming up with the so-called Doctrine of
Mutual Assured Destruction (MAD) that guided U.S. policy for many years.
The second was its role with the SAGE system for early detection and threat
assessment vs. an enemy approach from the Alaska direction. The third was the
movement from military arena to social programs. In the latter area, we find
RAND currently broken into nine divisions, with their scope implied by their
titles [7]:

• Homeland Security Operational Analysis Center

• Army Research Division
• Education and Labor
• Health Care
• National Security Research
• Social and Economic Well-Being
• Project Air Force
• Europe
• Australia

We proceed to the next chapter with an overview of the world of cost-effectiveness

analysis to which RAND contributed in a more-or-less direct way over the years.

REFERENCES
1. Enthoven, Alain, “How Systems Analysis, Cost-Effectiveness Analysis, or
Benefit Cost Analysis First became Influential in Federal Government Program
Decision-Making”, Cambridge Core, see https://www.cambridge/org/.
2. WSEIAC (Weapon System Effectiveness Industry Advisory Committee), Defense
Technical Information Center, 8725 John Kingman Road, Fort Belvoir, VA
22060-6218, #AD0467816.
3. Rosenzweig, P., “Robert S McNamara and the Evolution of Modern
Management”, Harvard Business Review, December 2010.
4. McNamara, Robert, The Fog of War, film and book.
5. Enthoven, Alain and Smith, K. Wayne, How Much Is Enough? RAND
Corporation, October 2005.
6. “The Whiz Kids: How 10 Men Saved America (and Then Almost Destroyed
It)”, see www.amnesta.net/other/whizKids/.
7. Automotive News, see www.autonews/com/article.
8. Google RAND Corporation, Los Angeles, CA.
An Overview of
the Basics 2
This chapter presents an overview of the basics of cost-effectiveness analysis
(CEA), as practiced by the author. The method is essentially one in which
comparisons are made between a set of well-defined alternatives. The costs
and effectiveness of each alternative is estimated. Absolute values are less
important than relative values for each alternative. Surrogates are used as
effectiveness measures, which are called measures of effectiveness (MOEs).
This method takes the point of view that there are many domains where one
may apply the CEA, such as transportation systems, warfare systems, and the
information system. Each domain has a history of experience that is called an
ensemble of systems. That history is represented by a large number of prior
instantiations where the data points are costs and MOEs. We can envision this
ensemble of systems and prior data points on a graph where the ordinate is
“effectiveness” and the abscissa values are costs.

SHAPE OF ENSEMBLE OF PRIOR SYSTEMS

The prior systems tend to form three basic regions, namely:
1. A linear region
2. A knee-of-the-curve region
3. A high-effectiveness region

The linear region has typically been found at low values of both cost and effec-
tiveness. Many systems have been developed in this area, which is a reflection of
both lack of funds as well as procurement practices. As we move up this linear
region, we reach a point where the curve leans toward the right. This means we
have to pay more and more for each increment of effectiveness gain. It is also often

DOI: 10.1201/9781003196174-2 13
14 Cost-Effectiveness Analysis

called the “best-value” region and we seek solutions in and around that region.
Finally, we move into the high-effectiveness region where we pay even more to
achieve greater levels of effectiveness. We generally do not like to be in this region
but accept the notion that many military systems are found in this region.

MEASURES OF EFFECTIVENESS (MOEs)

Various domains of systems were cited earlier. In this section, we are specific
about MOEs that tend to be used in a CEA. We list here some MOEs [1−3] for:

A. Transportation system
• Passenger capacity
• Freight capacity
• Distance/range
• Required power
• Storage capacity
• Speed/acceleration
• Capacity to demand ratio
• Braking distance
• Trip time
B. Communication system
• Speed of service
• Grade of service
• Detection probability
• False alarm probability
• Signal strength
• Noise power
• Signal-to-noise ratio
• Bit error rate (BER)
• Range
• Availability
C. Air defense system
• Detection probability of targets
• Number of targets in “track while scan” mode
• Number of missiles being tracked
• Threat assessment
• Absentee ratio
• Target kill probability
• Reliability
 2 • An Overview of the Basics 15

These are ways to ultimately evaluate the effectiveness of these types of

systems. They are not the elements of a system’s functional decomposition.
The MOEs for any particular CEA have to be selected by the user. On that
basis, we construct a matrix that contains the alternatives and a set of eva-
luation criteria for an illustrative intercity transportation system.

EVALUATION WEIGHTS ALTERNATIVE A ALTERNATIVE B ALTERNATIVE C

CRITERIA
RATING R x W RATING R x W RATING R x W
R R R
Trip time .2 6 1.2 7 1.4 8 1.6
Frequency of .1 7 .7 8 .8 9 .9
service
Pollution .1 7 .7 8 .8 9 .9
Convenience .2 6 1.2 7 1.4 8 1.6
Noise .2 7 1.4 8 1.6 9 1.8
Reliability .2 7 1.4 8 1.6 9 1.8
SUMS 1 6.6 7.2 8.6
(Effectiveness)

EFFECTIVENESS MEASURES
The aforementioned matrix shows a column called “SUMS,” which are the
effectiveness measures for the three systems A, B, and C. These are the sums
for the (R × W) (Rating times weighting) columns. The ratings are on a sale of
1 through 10, and are thus called surrogate measures. The three effectiveness
measures are thus:
OVERALL A B C
Effectiveness measures 6.6 7.2 8.6

If we increase the noise and pollution values, we obtain the new weights.
The matrix also shows the new effectiveness values based upon the increased
pollution and noise weights.
This method takes into account the possibility that the evaluation criteria
might have different degrees of importance. Thus, the weights are included,
16 Cost-Effectiveness Analysis

ALTERNATIVE ALTERNATIVE ALTERNATIVE

A B C
EVALUATION WEIGHTS RATING W x R RATING W x R RATING W x R
CRITERIA
Trip time 0.15 6 0.9 7 1.05 8 1.2
Frequency of 0.05 7 0.35 8 0.4 9 .45
service
Pollution .2 7 1.4 8 1.6 9 1.8
Convenience .15 6 0.9 7 1.05 8 1.2
Noise .3 7 2.1 8 2.4 9 2.7
Reliability 0.15 7 1.05 8 1.2 9 1.35
SUMS weights 1.0 6.7 7.7 8.7

with values that add to unity. A set of evaluation criteria and weights for a real
system evaluation are shown in Table 2.1. In this case, the criteria were de-
veloped from the inputs of the commissioners during an Aviation Advisory
Commission study [4].

TABLE 2.1 Values of weights for each commissioner and totals [ 4]

FACTORS 1 2 3 4 5 6 7 8 9 AVERAGE
Social effects 5 10 15 10 15 5 10 21 8 11.0
Environmental 20 40 10 15 20 5 15 8 12 6.1
Service quality 20 10 10 15 20 15 15 19 18 15.8
System capacity 10 10 10 15 20 20 15 15 13 14.2
Human factors 5 10 10 5 5 10 5 1 6 6.3
Internat’l economic 5 10 5 10 5 15 10 10 13 9.2
Investment costs 15 5 20 15 5 15 15 12 14 12.9
Operating costs 20 5 20 15 10 15 15 14 16 14.4

EVALUATORS (COMMISSIONERS)
We note that in this case, the commissioners insisted upon using some system
costs as evaluation criteria. This is distinctly not the way this author carries
 2 • An Overview of the Basics 17

out a CEA. However, this list reflects the way the commissioners wished to
look at the data.

SAMPLE COST DATA

We now assume some cost data for the use of three transportation systems, as
shown in the following table:
Cost of Cost of Cost of
Systems A System B System C
Round trip – use of $250 $350 $600
system

Is it clear which system a user will select for their intercity

transportation?

MOEs VS. FUNCTIONAL DECOMPOSITION

We take note of the difference here between MOEs and elements that com-
prise a systems functional decomposition. The former are selected so as to be
measurable, although the nature of that measurement may be difficult or less
than obvious. In the case of functional decomposition, we need to go there
when we are architecting a system.
An example of a functional decomposition of a generic information
system is shown as follows:

1. Input
2. Output
3. Processing
4. Operating system
5. Applications software
6. Security software
7. Database management system (DBMS)
8. Storage
9. Networking
10. Power supply
18 Cost-Effectiveness Analysis

A functional decomposition for a generic air defense system is as follows:

• Target surveillance
• Target detection
• Target tracking
• Missile assignment
• Missile firing
• Kill assessment
• Re-assignment of missiles
• Final kill assessment

Thus, we note that MOEs and functional decomposition elements for any
given system are quite different and should not be confused. Further, we take
as the basic form of the cost-effectiveness approach the weighting and rating
scheme presented here. Other variations on that same theme are suggested in
the Chapter 2 “An Overview of the Basics” on Miscellany.

SENSITIVITIES
There are several dimensions for the exploration of the sensitivities of the
effectiveness values. In general, as shown earlier, these dimensions are:

a. Selection of evaluation criteria

b. Estimated values for the evaluation criteria
c. Weighting factors
d. Estimated values for the weighting factors

Evaluation criteria – The criteria are critical to the overall evaluation and
need to be chosen very carefully. In a group process, special procedures are
needed for this selection.
Values of evaluation criteria – Values are assigned to each criterion on a
scale of 1 to 10. This has its arbitrary aspects to it, which remain a weakness
to the approach.
Weighting factors – This feature is important and is reflected in the
earlier Aviation Advisory Commission example.
Values for the weighting factors – Note the results for a group process in
which averages are calculated to obtain the final weighting factors.
 2 • An Overview of the Basics 19

Selection of Architectures vs. Selection of the

Best System to Use
Two perspectives are worth noting in terms of the use of cost and effec-
tiveness analyses. The first is with respect to architecting a system. In this
case, one is trying to find the “best” architecture for a given system. The
second has to do with selecting a best system to use. To illustrate, suppose
one is planning a trip, or series of trips, for the family. So questions of re-
levance appear to be:

• What is the best way for the family to travel back and forth from
home base (e.g., Washington, DC) to Los Angeles?
• What is the best way for the family to travel back and forth from its
home base to Miami, Florida?

We recognize that these kinds of questions have typically been the domain of
the very popular AAA service enterprise.
In the architecting domain, some relevant questions might be:

• What systems do we need to build to create the capability to travel

for Washington, DC to Los Angeles?
• What mix of systems do we need to build to create the capability to
travel from Washington, DC to Los Angeles?

This latter question might well require consideration of mixes of light rail,
high-speed rail, air travel, buses, as well as urban mass transit.

ABSOLUTE VALUES OF EFFECTIVENESS

MEASURES AND COST-RELATED VALUES
We take note of the estimation of effectiveness on a scale of 1 to 10 for each
alternative. We also cite this as a possible weakness of the overall approach. It
has the advantage, if you will, of allowing several measures to be “combined”
(i.e., to “add” effectiveness measures in order to obtain overall measures).
Another approach would be to attempt to calculate absolute values of each of
the given effectiveness measures. This was the approach the author’s team
study of intercity freight Some of the measures (not all) for that study in-
volved the following:
20 Cost-Effectiveness Analysis

(Nominal trip = 600 miles, 10 Mton system capacity)

Measure Typical Values for Five (*) Intercity Freight Transportation
Systems
FSI 3.2 times 107 (Freight Service Index)
NFSI 4.0 times 108 (Normalized Freight Service Index)
BTU/ton-mile 2,100
NFSI/pollutant 2 times 105
Viability Index −0.7

Cost-Related Measures
This list may be expanded to include cost-related estimates, as follows:

Total R&D costs 105

NFSI/R&D costs 3 times 103
Total system costs 10 times 109
NFSI/total cost 3.5 times 10−2
Return on investment (ROI) 1.0
Viability index −.075

(*) the five intercity freight systems were:

1. A 40-foot motor carrier

2. TRAILS, a conceptual high-speed freight system defined to be state
of the art
3. TOFC, a trailer or flat car dedicated 70-car train
4. NB Igloo, a narrow body containerized air-freight system
5. WBLH, a wide-body lower-hold containerized air-freight system

We note that these measures are selected independently and, in general, have
the property that they are orthogonal to one another.

THE REAL MEANING OF COST

ANALYSIS [4]
Some cost information is appropriate here as we close this chapter. Back in
1983, D. Novick (affiliated with RAND) gave a talk for the Institute of Cost
 2 • An Overview of the Basics 21

Analysts [4]. This illustrated how cost analysis can (and should) be part of
systems analysis. In that situation, the author cites the problem as deciding
which solution is best in a selection of a turbo-prop bomber as opposed to a
pure jet. The cost data came from the Comptroller at Headquarters USAF.
Going into a critical meeting in Dayton, the Air Force was prepared to an-
nounce the winner. However, once they looked at the “new” cost data, they
found that the price of the turbo-prop had doubled and the price of the pure jet
had gone down by some 50%. That was enough to flip the decision and
convince all concerned that RAND needed its own way of gathering reliable
data. So, in that example, cost was a major driver, and well that it was. A
reminder – cost-effectiveness has two components, not to be forgotten.

THE BOTTOM LINE

This method of cost-effectiveness analysis may be considered a systems
approach to the field in that it is largely derived from systems analysis and
architecting. We use a systems perspective that is larger than simply defining
alternatives. This leads, most of the time, to three or more alternatives. It also
leads to looking at multiple systems above and beyond the attributes of only a
couple of alternatives. We also note that for the DoD’s approach to AoA, a
minimum of three alternatives are called for [5]. This is viewed as corro-
borating evidence in favor of the three-alternative approach to both archi-
tecting and cost-effectiveness analysis.

REFERENCES
1. Eisner, H., Systems Engineering – Fifty Lessons Learned, CRC Press, 2021.
2. Aviation Advisory Commission (AAC) Study.
3. Eisner, H., Computer-Aided Systems Engineering, Prentice Hall, 1988, Intercity
Transportation Study, page 250.
4. Novick, D., “Meaning of Cost Analysis”, Luncheon Presentation, LA Chapter of
Institute of Cost Analysis, May 18, 1983.
5. AoA Approach, DoD, see Wikipedia.org/Analysis of Alternatives.
Cost-Effectiveness
and Systems
Engineering
3
OVERVIEW
This book has the words “a systems engineering perspective” in its title. It is
the purpose of this chapter to explain the meaning of this phrase. First, we can
say that systems engineering is not a part of cost-effectiveness analysis, but
the reverse is true, that is, cost-effectiveness is a part of systems engineering.
And we will define here just exactly how that is the case. We start that by
various comments about systems engineering.

WHAT IS SYSTEMS ENGINEERING?

A good place to begin in this regard is with a standard known as 15288 [1].
This important standard marks the systems engineering life-cycle processes.
These processes are the main “tasks” of systems engineering. These are the
main “ingredients” of systems engineering and are the critical “activities” of
systems engineering that the International Council of Systems Engineering
(INCOSE) insists that all systems engineers need to master and be able to
apply to the real-world building of systems.
As stated earlier, cost-effectiveness is a part of systems engineering. For
this author, this part is very well-defined and specific and is the “architecture
definition process” [2] whose purpose is stated as:

DOI: 10.1201/9781003196174-3 23
24 Cost-Effectiveness Analysis

The purpose of the architecture definition process is to generate system architecture

alternatives, to select one or more alternative(s) that frame stakeholder concerns and
meet system requirements, and to express this in a set of consistent views.

For this author, cost-effectiveness analysis is the way in which one

carries out the architecture definition process.
So that might imply that without cost-effectiveness analysis the systems en-
gineer cannot define a system architecture. Such is the appearance, but not ne-
cessarily the case. It turns out that there are other ways to carry out the architecture
definition process, the major one being the DoDAF [3] approach. However, this
author suggests that the best way to carry out the architecture definition process is
to use cost-effectiveness analysis. Subscribing to this approach means that there is a
placeholder for cost-effectiveness analysis in each and every system that is being
built. It means that each and every time we are confronted with the architecture
definition process we move forward with the cost-effectiveness analysis method.
And such will be the case until and if a better method can be found.
Two other comments appear to be relevant at this point in the discussion.
The first has to do with the SEBoK; the second relates to the formal Analysis
of Alternatives (AoA) as articulated by the Department of Defense (DoD) and
the MITRE Corporation.

THE SEBoK
Yet another source of definitive information regarding systems engineering is
the Systems Engineering Body of Knowledge (SEBoK) [4] The initiative for
the SEBoK appears to lie with the Stevens Institute of Technology. It is a
compendium of facts about systems engineering that represents, as the name
implies, the basic body of knowledge.
Version 2.3 of SEBoK was released in October 2020. The essence of
SEBoK includes the following eight parts:

a. SEBoK Introduction
b. Foundations of Systems Engineering
c. Systems Engineering and Management
d. Applications of Systems Engineering
e. Enabling Systems Engineering
f. Related Disciplines
g. Systems Engineering Implementation Examples
h. Emerging Knowledge
 3 • Cost-Effectiveness and Systems Engineering 25

Emerging aspects of systems engineering are considered to be such topics as

systems of systems, agile life cycle approaches, and model-based systems
engineering. This author commends the Stevens Institute of Technology for
its role in developing and maintaining the SEBoK.

ILLUSTRATION OF COST-EFFECTIVENESS
APPROACH TO ARCHITECTING
This is an example of using the cost-effectiveness approach to architecting a
system. The four key steps in architecting a system are [5]:

a. Functional decomposition
b. Synthesis
c. Analysis
d. Cost-effectiveness considerations

The second step is the process for a hypothetical information system. The
three systems that are being compared are characterized as (A) the Dell/
Access system, (B) the Lenovo/SAS system, and (C) the Workstation/
Oracle system.

EVALUATION ALTERNATIVE SYSTEMS

CRITERIA
A B C
WEIGHTS DELL / ACCESS LENOVO / SAS WORKSTATION /
ORACLE
R WxR R WxR R WxR
Overall risk .2 .7 .14 .8 .16 .9 .18
Ease of .2 .8 .16 .9 .18 .95 .19
implementation
Rejection of .1 .75 .075 .85 .085 .9 .09
malware
Performance .3 .7 .21 .9 .27 .95 .285
Maintainability .2 .8 .16 .85 .17 .95 .19
TOTAL 1.0 .67 .865 .925
where R = rating and W = weight.
26 Cost-Effectiveness Analysis

We note that the analysis step sets forth a series of evaluation criteria,
namely (a) overall risk, (b) ease of implementation, (c) rejection of (all types
of) malware, (d) overall performance, and (e) maintainability. These criteria
are given the weights, and they add to unity. Each system is then rated on a
scale of one to ten, for each criterion. The weights times the ratings are
calculated as are the sums down the three systems columns. The totals are
taken to be the overall effectiveness measures for each of the three systems.
They are:

Effectiveness (A) = 0.67

Effectiveness (B) = 0.865

Effectiveness (C) = 0.925

With this procedure, we are, by definition, moving forward with choosing an

architecture for the IT system in question. We now continue the procedure by
adding independent cost estimates for these three systems, as follows:

Cost (A) = $20 million

Cost (B) = $40 million

Cost (C) = $70 million

ANALYSIS OF ALTERNATIVES (AoA) [6]

The Department of Defense has contributed to this field by sponsoring what
they have called the AoA. They have mandated such for their high-value
and important systems [7]. Indeed, they have also specified that an AoA
shall include a minimum of three alternatives in the analysis. Curiously, the
cost-effectiveness approach recommended by this author suggests three
alternatives, approaching the problem from a different direction and also
many years of experience and students’ examples. The reader would do
well to keep the three alternatives approach in mind as it is reinforced by
two sources.
 3 • Cost-Effectiveness and Systems Engineering 27

The three alternatives are compared using such metrics as technology ma-
turation, key performance parameters (KPPs), suitability, operational effectiveness,
and risk. Performing an AoA is mandatory within the DoD for certain classes of
systems, and the policy is controlled by the well-known Office of Management and
Budget (OMB). The needs, as represented in the AoA, are derived from the Joint
Capabilities and Integration Development System (JCIDS). The listed objectives of
the AoA are [6]:

a. To refine the alternatives

b. To refine the evaluation criteria
c. To refine the evaluation process
d. To work toward obtaining consensus
e. To reduce overall uncertainty
f. To choose one alternative (from the three)

MITRE [8,9] defines AoA as “an analytic comparison of the operational ef-
fectiveness, suitability, risk, and life cycle cost of alternatives that satisfy
validated capability needs.”
As such, the AoA documents the rationale for one alternative selection
above and beyond the other alternatives. MITRE also cites the typical steps in
an AoA as:

a. Plan
b. Establish analysis foundation
c. Identify and define alternatives
d. Assess alternatives
e. Compare alternatives
f. Report results

The Office of Aerospace Studies [8] explains AoA as the focus of the Concept
and Technology Development Phase of the systems in question. This source
cited the following as objectives of the AoA. The reader is urged to compare
this with the MITRE approach mentioned earlier.

a. Refine alternatives
b. Refine criteria
c. Refine evaluation
d. Work to gain consensus
e. Reduce uncertainty
f. Choose an alternative
28 Cost-Effectiveness Analysis

MITRE’S SYSTEMS ENGINEERING GUIDE

We complete this chapter with two perspectives, one from MITRE and the
other from INCOSE’s systems engineering handbook. MITRE has been one
of this nation’s most-accomplished systems engineering enterprises. MITRE
came into being in 1956 supporting the Air Force Electronic Systems
Division at Hanscom Field in the Boston area. Since that time it has expanded
as an Federally Funded Contract Research Center (FCRC) into new areas of
research and also new locations.
The reference here is from the MITRE Systems Engineering Guide [10]
in which they emphasize affordability, efficiency, and effectiveness (in dis-
tinction to, for example, cost-effectiveness). They claim that these three
features “guide systems engineers in developing and shaping engineering
solutions, making program recommendations, and evaluating engineering
efforts.” This is followed by more precise definitions of affordability, effi-
ciency, and effectiveness. As far as MITRE is concerned, the connection is
between their roles as systems engineers and these three features, representing
a step beyond, but including, cost-effectiveness. It’s a perspective that likely
came from their unique consulting positions on a variety of programs. And it
seems to be an extension of the pure evaluation of alternative systems on a
cost-effectiveness basis.

INCOSE SYSTEMS ENGINEERING

HANDBOOK [11]
Another important connection between systems engineering and cost-
effectiveness can be seen directly in the INCOSE systems engineering
handbook. First, this source points out that the systems engineer cannot afford
to be ignorant of system costs as well as its cost-effectiveness. It goes on to
declare that the systems engineer, at the systems level, should use cost-
effectiveness analysis (CEA) to “derive critical system requirements and
support data-based decision-making.” It clearly reinforces the connection
between the two disciplines (i.e., systems engineering and cost-effectiveness
analysis) by giving examples of how cost-effectiveness is used in various
planning and management activities. Topics cited include, but are not lim-
ited to:
 3 • Cost-Effectiveness and Systems Engineering 29

a. Studies of aircraft performance

b. Urban studies of a city’s transportation features
c. Health services
d. The acquisition of military hardware
e. The life cycle cost (LCC) of one or more systems

The latter item is shown to include the following:

1. Concept and development costs

2. Production costs
3. Utilization and support costs
4. Retirement costs

All of the aforementioned items are covered within the overall topic and
chapter dealing with specialty engineering activities. Much of the cost ana-
lysis is thought of as a subset of the term known as affordability. So it’s a
matter not only of what the costs are, or might be, but of whether or not these
costs are “affordable.”

REFERENCES
1. ISO/IEC/IEEE 15288, Systems and Software Engineering – System Life Cycle
Processes, ISO, 2015.
2. David Walden, et al., Systems Engineering Handbook, Fourth Edition,
INCOSE, John Wiley, 2015.
3. DoDAF, v. 2.02, August 2010, DoD, OSD, Washington, DC.
4. SEBoK, “Systems Engineering Body of Knowledge”, see www.sebokwiki.org.
5. Eisner, H., Systems Architecting, CRC Press, 2019.
6. Analysis of Alternatives (AoA), see en.wikipedia.org/Analysis of Alternatives.
7. DoD AoA, DoD Instruction 5000.02, January 23, 2020.
8. See MITRE Report, Analysis of Alternatives, Systems Engineering Guide, 1997.
9. Analysis of Alternatives Handbook, Office of Aerospace Studies, Air Force
Material Command, June 2013.
10. See MITRE Systems Engineering Guide: Affordability, Efficiency and
Effectiveness, 1997.
11. INCOSE Handbook, see Reference [2].
Selected
Commentaries on
Cost-Effectiveness
4
Although cost-effectiveness is not an especially controversial topic,
there has been quite a lot of commentary about it over the years. This
chapter covers some of this commentary. First, we examine a survey
of weapon system cost-effectiveness methodologies [1]. This author
articulates some four reasons that cost-effectiveness estimates are used,
namely:

1. To provide objective criteria in order to select a preferred system

2. To highlight technical and cost weaknesses of the alternatives
3. To set the stage for moving into the Contract Definition Phase for
the system
4. To define the traceability of important system performance parameters

Three levels of analysis are presented:

a. Overall defense goals (Decision-Making Level)

b. System competition (First Level Models)
c. Component selection (Second and Third Level Models)

The author summarizes by citing some of the features and systems that belong
in each category.

DOI: 10.1201/9781003196174-4 31
32 Cost-Effectiveness Analysis

McNAMARA AS A TRANSFORMATIVE
LEADER [2]
The Colonel’s dissertation emphasizes the transformative effects brought
about by Robert McNamara as he installed the PPBS within the Office of the
Secretary of Defense (OSD) during the Kennedy–Johnson years. The author
claims that McNamara disregarded many of the approaches to contemporary
management that included delegation, flexibility, and informal communica-
tions. Instead, he was personally involved in the analysis of data and various
forms of quantitative analysis, to include cost-effectiveness analysis. As a
consequence, he often overlooked the “big picture,” which at that time was
the war in Vietnam.
McNamara and Kennedy agreed on a new defense orientation. The old
one, which was rejected, was a first strike attack. The new one was to em-
phasize the need for adequate strategic arms and defense to deter nuclear
attack on the United States and its Allies. A 1961 review led to the decision to
increase the nation’s limited warfare capabilities. That review also confirmed
the need for the United States to be able to counter possible Communist wars
of “national liberation.”
Col. Lippert winds up supporting the work and style of McNamara,
pointing out that McNamara became disenchanted with the Vietnam War
outcome, and moved on to become the President, in 1967, of the World Bank.
He quotes Former Secretary of State Dean Acheson as saying:

except for General Marshall, I do not know of any department head who,
during the half century I have observed government in Washington, has so
profoundly enhanced the position, power and security of the United States,
as Mr. McNamara.

He also quotes journalist Hanson Baldwin’s [2] list of impressive accom-

plishments, to include:

a. Containment of negative service rivalries

b. Curtailment of duplication and waste in weapon procurement
c. Installment of the planning, programming, and budgeting (PPBS)
system; systems analysis; and analytic tools, such as cost-effectiveness
analysis
d. Expanded use of computer technology
e. A flexible strategy, which improved the U.S. capability to wage
conventional and limited warfare
 4 • Commentaries on Cost-Effectiveness 33

A HISTORY OF COST-EFFECTIVENESS
ANALYSIS
We next move on to a history from E. S. Quade of the RAND Corporation in
1971 [3]. This paper starts by detailing the merits and lack thereof of the cost-
effectiveness of the Safeguard Program. The author points out that we can
decide between competing alternatives but not determine the best overall use
of our money. Three specific examples from history are cited in some detail,
leading to the 1950s. After WWII, we see the use of operations research and
systems analysis that included cost-effectiveness analysis. A boost for cost-
benefit analysis was provided when President Johnson mandated the use of
PPBS activity across the board in the government, through the Bureau of the
Budget.
By way of modern development, Quade goes on to trace cost-effectiveness
to three “streams of development,” namely economic theory, practical en-
gineering, and the operational analysis of WWII.
Specific criticisms appeared in McKeon’s 1958 book [4] in the form of
C/E ratios, and other omissions and biases. Overall conclusion – despite the
deficiencies, the future is sound and bright for cost-effectiveness analyses.

A CRITIQUE OF COST-EFFECTIVENESS
In November of 1975, Quade produced his critique of cost-effectiveness [5].
The key comment in this critique may be quoted as:

the defects I find in cost-effectiveness – that price is equated to cost; that

effectiveness is equated to value; and that cost and effectiveness together are
assumed to be all one needs to know to choose among alternatives.

He goes on to say that effectiveness doesn’t necessarily measure value and

that cost is more appropriately the loss or penalty incurred by selecting an
alternative. Quade goes on to say that “we need set the scale of the effort –
either a cost we must not exceed or an effectiveness level we must achieve.”
These penetrating comments were registered some 40 years ago, and stand in
good stead today. Mr. Quade is well worth reading over and over again, as
well as between the lines.
34 Cost-Effectiveness Analysis

PRINCIPLES AND CONTROVERSIES [6]

The source here is a set of slides used by Emmett Keeler (RAND), in 1994, to
discuss some cost-effectiveness principles and controversies. The author
presents a matrix summary of effects for a three-option comparison of the
following variables:

a. Direct cost
b. Indirect costs
c. Years of life experience
d. Restricted activity in days
e. Level of satisfaction

The claim is that this is typically Table 2 of any CEA paper. The author notes
that it is useful to try to aggregate all the effects into a single dimension, in
dollars. From there, calculate a cost-effectiveness ratio. For this author, the
presentation is clear but still difficult to follow.

THE OVERALL ATMOSPHERE IN THE

OSD [7]
The centerpiece of decision-making in the OSD, of course, was the Secretary of
Defense. As strong as McNamara was, he needed a lot of help. He turned for
that help, in the main, to his Assistant Secretary of Defense for System
Analysis, Alain Enthoven. He also got help, depending upon the issue, from the
individual services and advocates therein. There was an ever-present tension,
however, between the “old-time military” folks who had yards and yards of
operational experience in one or the other of our armed forces. To name a
couple, we have Curtiss LeMay and Hyman Rickover. The arm-chair analysts,
as smart as they were, could not see people like LeMay and Rickover, or their
staffs, as pushovers. The battles were heavy duty, and at times over the edge.
Here’s the “story” on the Admiral Rickover comment. Apparently, in
1968, before a congressional committee, Rickover commented as follows:

the social scientists who have been making the so-called cost-effectiveness
studies have little or no scientific training or technical expertise; they know
little about naval operations …. Their studies are, in general, abstractions ….
 4 • Commentaries on Cost-Effectiveness 35

In my opinion, we are unwise to put the fate of the United States into their
inexperienced hands. If we keep on this way, we may find ourselves in the
midst of one of their cost-effectiveness studies when, all of a sudden, we learn
that our opponents that are faster or better than ours.

Yet another perspective on Rickover’s thinking with respect to systems

analysis is revealed in Rickover’s biography [8,9]. In this treatise, it is de-
clared that McNamara thought that the Polaris missile submarines were not a
Navy Program, but rather a strategic program. Thus, they were to be con-
sidered along with ICBMs and manned bombers but not alongside of attack
submarines and amphibious ships. Alain Enthoven elucidated by saying that
“nuclear power for surface ships offers a major increase in effectiveness.” He
went on to say that this increase needed to be quantified so that the appro-
priate final evaluation could be accomplished.
This type of point was made over and over again as part of the analysis of
alternatives – namely, what exactly are the alternatives? This was to be the
subject of many a controversy over how to proceed with a budget as part of
the overall PPBS in the OSD.
Tapgaard describes some of the overall atmosphere in the OSD [7]. The
rational analysts came to the table with their cost-effectiveness analyses, but it
often did not carry the day on important decisions and systems. It only served to
give cost-effectiveness a bad name which, in all likelihood, it did not deserve.
Nonetheless, here was a classic struggle that, day by day, yielded a defense plan
and budget. One could be sure, during McNamara’s tenure as SecDef, that
highly complex and sophisticated analysis methods were brought to bear and to
the advantage of decision-makers in this very important part of our government.
Tapgaard emphasizes, as examples, cases related to:

a. Precision Guided Munitions

b. The TFX Program
c. The CVA Program

These examples demonstrate both successes and what might be considered

setbacks by McNamara during his long tenure (seven years) as SecDef.

SELECTED COURSES IN COST-EFFECTIVENESS

It appears that relevant courses have maintained a high level of interest in the
academic world. Note the sampling of courses below from (A) the University
36 Cost-Effectiveness Analysis

of Maryland, (B) Harvard, (C) The George Washington University, (D) the
University of Chicago, and (E) the Pardee RAND Graduate School.

A. University of Maryland
a. A specific course is provided that gives students the knowledge
and skills to do program evaluation using examples from do-
mestic policy and international development. The course in-
cludes cost-effectiveness and cost-benefit studies from
performance measurement of ongoing programs.
B. Harvard – The Harvard Center for Health Decision Science
emphasizes:
a. Overall approaches
b. What is decision science?
c. Practice and policy
d. Cost-effectiveness analysis
e. Models and tools
f. Preferences and values
g. They also have reference case guidelines for Benefit-Cost
Analysis I B, Global Health and Development, and a Journal
of Benefit-Cost Analysis
C. The George Washington University (GWU) – The GWU course is
designed around the interests of MPP and PhD Public Policy and
Administration students. There are extra readings for PhD students
and extra reading for MPP students. These readings will generally
allow the students to:
a. Conduct a thorough cost-benefit analysis
b. Understand the limits of the aforementioned analysis
c. Analyze cost-benefit analyses from others
d. Apply economic theory to real-world policy problems
e. Compare alternative policies using the tools from the fields of
cost-benefit and economics
D. University of Chicago – This school emphasizes a cost-benefit
analysis with the following course description:

Specific attention is paid to the goals of the course include learning (1)
how to read a cost benefit analysis (CBA), (2) how to bring CBA work
into an overall policy statement, and (when CBA is a good tool and when
it is not. The course also deals with uncertainty and risk, discount costs
and benefits received into the future, the value of lives saved, and the
overall CBA methodology. To include the microeconomic underpinnings
 4 • Commentaries on Cost-Effectiveness 37

of the technique, and applications drawn from various areas, such as

health, the environment and public goods and services.

E. Pardee RAND Graduate School – This school has a course that

introduces two analytic tools that can be used to structure problems
and perform both cost-benefit and cost-effectiveness analyses. The
tools apply economic methods to carry out a more efficient alloca-
tion of resources where markets are not working well. Cost-benefit
analysis is used in fine-tuning public investments and regulations. It
converts all the estimated inputs and outcomes into current dollar
equivalents. Cost-effectiveness is used whenever all outcomes can
be expressed in one non-financial metric that has been used com-
monly in health care.

Generally, graduate courses cover cost-effective analysis, cost-benefit ana-

lysis, and economic analysis as well as other specialized courses (often in the
health domain).
Cost-effectiveness tends to support positive programs, and conversely. It
remains a powerful tool for decision-making up through the Obama
Administration. Unfortunately, it was more or less abandoned during the
Trump tenure.

A BOTTOM LINE FOR COST-EFFECTIVENESS

AND COST-BENEFIT ANALYSES
A perspective for the difference between cost-effectiveness and cost-benefit
analyses completes this chapter. As presented here, for cost-effectiveness,
we define, for example, three systems to serve a particular purpose, or
provide a set of functional capabilities. For each such system, we calculate
the costs and the effectiveness, where the effectiveness are based upon a
series of weights and evaluation criteria. We are interested in the three re-
gions of ensembles of systems:

a. The low-cost-low-effectiveness region

b. The knee-of-the curve region, and the
c. High-effectiveness region

We select the “best” system of the three that have been defined and evaluated.
38 Cost-Effectiveness Analysis

For cost-benefits, we generally tend to define a “program” for the public

benefit. As an example, such a program might be a health program for the
public, such as “Obamacare.” The defined program has calculable benefits that
are enumerated in detail and put into monetary terms. The costs for such a
program are then calculated and compared to the benefits. If the benefits exceed
the costs, we conclude that the program is cost-beneficial, and support the
implementation of such a program. If the costs exceed the benefits, we tend to
not support the program, and look for ways to increase the benefits without
increasing the costs. This can be, and often is, a difficult design problem.
For the latter, we need to be sure that the benefits are provided to the
same population that bear the costs. A program that will not work, for ex-
ample, is one for which the top 10% of earners gain the benefits but it is paid
for by the entire public. Here again, these are difficult and often subjective
calculations with considerable uncertainty embedded in them. But this last
principle is an important one.

OPEN-ENDED PROGRAM DESIGN AND

ANALYSIS [10]
Choosing among alternatives has been the theme for both cost-effectiveness
and even cost-benefits, though more strongly for the former. How the alter-
natives are defined and developed is a bit of a mystery, though we seem to
have little trouble with its accomplishment. However, there seems to be a lot
of room within the Department of Defense to simply define a worthy task and
just plan go out and do it. When the DoD decides on such a task, there is
usually enough time, money, and clout to move forward without requiring a
cost-effectiveness analysis. One such example is discussed later, completing
this chapter. It has the name – open-ended program design and analysis.
In March 2018, the then-Deputy Secretary of Defense (Shanahan)
decided that we needed to take a deeper look at how to build a more resilient
space architecture that would be able to outpace any and all emerging threats.
Without further fanfare or cost-effectiveness analysis, he instructed the
Aerospace Corporation to conduct such a study. The effort was called Project
Thor, and it resulted in a report as listed here [11].
The National Security Space Enterprise explored a Continuous
Production Agility (CPA) approach that would deliver an entire constellation
over a short period of time, say, five years. When approved, that would lead to
a replenishment activity on a definitive schedule. Within the DoD, we can be
 4 • Commentaries on Cost-Effectiveness 39

assured that many such studies have been defined and carried out over the
years, using the good offices of the FFRDC known as the Aerospace
Corporation. Other implementors include the MITRE Corporation. The
“umbrella” for such a set of activities, in broad terms, is the PPBS, a system
that gives the SecDev the flexibility to program and re-program funding to
accomplish short-term, long-term, and pop-up studies. So, we can be assured
that the hands of the many colonels in the Pentagon are not idle – they’re
responding to one thing or another (like requests from the legislative branch
or Pentagon higher-ups, and they get all of it done.

ANOTHER LOOK AT DEFINING

ALTERNATIVES
We take it for granted that cost-effectiveness analysis is directly applicable
when we are comparing alternatives. We are generally suggesting three al-
ternatives, for reasons cited earlier. But we need to take a deeper look at how
these alternatives come to be.
One instance that can be examined is when McNamara was briefed that
the Air Force and the Navy both added tactical aircrafts into their budgets for
the year. He challenged this approach, questioning as to the possibility of a
single aircraft for both services. He was counter-challenged by both services,
arguing that the missions were different and so was the need for two different
aircraft. This became a serious battle over a period of time, going way beyond
a “simple” cost-effectiveness analysis.
A considerably less difficult way to define alternatives is to restrict one’s
look at commercially available products. In such cases, several searches on
your reliable google website will reveal many alternatives, and their features.
As examples, take a look at the following lists for computers, telephones, and
automobiles.

Computers
• Lenovo
• Microsoft Surface
• HP Pavilion
• Apple iMac
• Dell
• Samsung Chromebook
40 Cost-Effectiveness Analysis

Telephones
• Samsung Galaxy
• Google Pixel
• Ooma Yealink
• Panasonic
• Alcatel

Automobiles
• Toyota
• Honda
• Nissan
• Ford
• General Motors
• Subaru

RELATIONSHIP BETWEEN
COST-EFFECTIVENESS AND
SYSTEMS ENGINEERING
Finally, we reiterate the fact that cost-effectiveness analysis and systems
engineering are intimately related (see previous chapter). For this author,
indeed, cost-effectiveness analysis makes a critical contribution to a crucial
part of systems engineering. This latter, of course, is a discipline that allows
us to build successful systems. A central feature is that of designing such a
system, and in particular, architecting such a system. This author has set forth
a method of architecting [12] that embodies cost-effectiveness analysis.
Although this is not the only connection between cost-effectiveness and
systems engineering, it is the deepest connection. An overriding consideration
is also that both follow the “systems” approach [13].

REFERENCES
1. Fritz, J. T., “A Survey of Weapon System Cost Effectiveness Methodologies”,
University of Central Florida, STARS Program, 1976.
 4 • Commentaries on Cost-Effectiveness 41

2. Lippert Col., A. J., “McNamara as a Transformative Leader”, Strategy

Research Project, see https://www.jstor.org.
3. Quade, E. S., “A History of Cost-Effectiveness”, IFORS International Cost-
Effectiveness Conference, April 1971, Washington, DC.
4. McKeon, R., Efficiency in Government through System Analysis, John
Wiley, 1958.
5. Quade, E. S., “A Critique of Cost-Effectiveness”, session on Cost-Effectiveness,
ORSA/TIMS Las Vegas, NV, November 1975.
6. Keeler, E. “Cost Effectiveness Principles and Controversies”, June 1944,
AHSR Annual Meeting.
7. Tapgaard, P. T., “McNamara and the Rise of Analysis in Defense Planning: A
Retrospective”, Naval War College Review, Vol. 48, Number 4, Autumn 1995.
8. Polmar, N., and Allen, T., Rickover – Controversy and Genius, A Biography, A
Touchstone Book, 1982.
9. Fox, P., “A Theory of Cost Effectiveness for Military Systems Analysis”, April
1965, see https://pubsonlineinforms.org.
10. “Project Thor – Continuous Production Agility (CPA) – Adapting at the Speed
of Relevance”, Aerospace Corporation, 2020.
11. “Outpacing the Threat with an Agile Defense Space Enterprise”, Aerospace
Corporation, 2020.
12. Eisner, H., Systems Architecting – Methods and Examples, CRC Press, 2020.
13. Eisner H., Systems Engineering – Building Successful Systems, Morgan and
Claypool, 2011.
Life Cycle
Costing 5
This chapter deals with one-half of the cost-effectiveness “equation,” namely,
life cycle costing. This will be embodied in what this author will call a life
cycle cost model, which will define all the cost elements that generally need
to be considered, over a period of 20 years. Other cost “factors” that are
relevant to the overall subject are cited in later chapters.
The Defense Acquisition University (DAU) [1] indicates that all DoD
agencies submit life cycle cost estimates (LCCEs) when requested, from
program initiation to disposal. These inputs serve two purposes:

a. Support the premise of affordability, and

b. Form the basis for a budget request from Congress

The cost structure is model based, with the five top-level categories of cost:

1. R, D, T, & E
2. Procurement
3. Operations and Maintenance (O&M)
4. Military Construction (MILCON)
5. Military Personnel (MILPERS)

A similar approach is spelt out in the DoD Cost Analysis and Procedures, but
not with these exact categories. Instead, they are:

1. Research and Development (R&D)

2. Investment
3. Operating and Support (O&S)
4. Disposal

DOI: 10.1201/9781003196174-5 43
44 Cost-Effectiveness Analysis

GENERAL CATEGORIES OF COST

The top-level categories of cost, as accepted in this book, and not with-
standing the aforementioned categories, are:

1. R, D, T, & E (Research, Development, Test, and Evaluation)

2. Procurement
3. Operations and Maintenance (O&M)

Over time, we have accepted these categories and kept track of historical
costs. Detailed cost elements in these three categories are cited in the fol-
lowing sections [2].

SUBORDINATE DETAILED COST

ELEMENTS
The subordinate detailed costs for each of the top-level categories of cost are
cited as follows:

1.1 Research and Development (R&D)

Preliminary Studies
Design Engineering
Hardware
Software
Other Personnel Costs
1.2 Test and Evaluation
Test Planning
Test Hardware
Test Software
Test Operations
Test Evaluation
Other Personnel Costs
 5 • Life Cycle Costing 45

2. Procurement
2.1 Installations
New Construction
Modification and Renovation
2.2 Equipment (Hardware and Software)
Primary Mission
Mission Support
Other Specialized
2.3 Stocks
Initial Stock (Primary Mission)
Initial Stock (Support Mission)
Spares (Primary and Support)
2.4 Initial Training
Training and Support Personnel
Training Materials and Equipment
Training Facilities
2.5 Other Procurement (e.g., Transportation) Costs
3. Operations and Maintenance (O&M)
3.1 Equipment Replacement (Hardware and Software)
Primary Mission
Mission Support
Other Specialized
3.2 Maintenance
Primary Mission
Mission Support
Other Specialized
3.3 Training
Training and Support Personnel
Training Materials and Equipment
Training Facilities
3.4 Salaries (Operators)
System Operators
Other Operational Support
3.5 Material
Expendables
Other Support Material
3.6 Other Operations and Maintenance Costs (e.g., Transportation)

When we refer to a “life cycle cost” model, we are considering all of the
aforementioned costs over the life of a system. Typically, the system life is
some 20 years so we can envision a spreadsheet with the aforementioned
46 Cost-Effectiveness Analysis

elements as rows and 20 years as the columns. That constitutes the life cycle
cost “model.”

OTHER COST CATEGORIES

There are various enterprises that look at costs in terms of budgets, from
today to historical. To illustrate this approach, we see an analysis of DoD
budgets [3,4] that breaks costs down into the following categories:

a. R, D, T, & E
b. Procurement
c. Operations and Maintenance
d. MILPERS
e. MILCON
f. OCO

Where MILPERS is military personnel, MILCON and OCO refer to Military

Personnel, Military Construction and Overseas Contingency Operations, re-
spectively [3].

Difficulties in Obtaining Real Cost Data [3]

As with any model, one must be careful to make sure the data is real and
comprehensive. Here are some thoughts suggested by a cost analyst at the
RAND Corporation:
Using Multiple Data System Products. This typical problem derives
from using multiple data sources that may not be compatible.
Different Data Nomenclatures. Different names for same or related
costs.
Insufficient Data Quality and Quantity. Derives from poor data col-
lection and verification.
Suggestions for getting around difficulties
Establish New Data Collection Procedures. Start with new clean sheet
of paper approach, to include double checking and verification.
Cite All Data Sources. All Data must be traceable to the source of the
data, which must be readily available.
Use Special Database for LCC Analysis. Similar to the earlier point,
but with support from new budget entry.
 5 • Life Cycle Costing 47

Go to System Advocates for Common Approach. System advocates

have special interest in obtaining good data. Assure, where possible, data
sources using same approach; work with them to achieve.

THREE TOP CATEGORIES OF COST IN

RELATION TO LIFE CYCLE STAGES
Of course, we expect to see the top-three categories of cost mapped differ-
ently across the life cycle stages of any system. For example, the R, D, T, & E
costs are essentially zero during the conceptual phase and are quite small
during the design validation phase. They go into high gear as we enter the
development and prototype testing phase, as one might expect. The pro-
curement costs are generally associated with entering the production and
acquisition phase. That phase triggers the operational phase with the cost of
ownership and O&M expenditures. When the R, D, T, & E phase is fully
complete, one enters the salvage phase. So, the system’s planner needs to
program the three types of costs into the system’s life cycle stages, however,
they may be defined at that time.

NIST/GSA BUILDING HANDBOOK [5]

The following are some guidelines for LCC in the context of building design.

a. An acceptable level of building services must be assured.

b. Design alternatives must be compared against a baseline that is the
lowest first cost under consideration.
c. The analysis period should be chosen to fully represent all costs.
d. Costs already incurred are considered “sunk” costs and are ex-
cluded from the model, but cited and noted.
e. Baseline and alternative first costs are estimated for the construc-
tion award date.
f. Salvage values for alternatives are typically zero.
g. Formulas listed in the reference are available as per the following
citations:
1. Sunk costs – example is a design fee – not included in the analysis
2. First costs – investment costs, construction costs, purchase
price, and present value
48 Cost-Effectiveness Analysis

3. Salvage value – scrap value of equipment, value at end of

service life
4. Future investment – one-time investments after start of analysis
period
5. Residual value
6. Annually recuring fixed value
7. Annually recurring escalating value
8. Energy expense – by month and year and total

DoD INSTRUCTION 5000.73 – COST

ANALYSIS GUIDANCE [6]
Salient points made by the DoD with respect to cost analysis and guidance for
2020 are summarized as follows:

• These points apply to making cost estimates for major defense

acquisition programs (MDAPs) and major automated information
systems (MAIS) at key acquisition events.
• Timelines are cited as to when cost estimates are provided.
• The estimates of cost are to be provided in a cost assessment and
program evaluation (CAPE) annual report
• The articulation of methodology is provided under the subjects:
a. Fleet alternatives
b. Airlift requirements
c. Effectiveness of alternatives
d. Cost estimates
e. Cost and effectiveness values to facilitate decision-making.

DoD BUDGET SPENDING BY

CATEGORY [3]
Budget data for three fiscal years follows to give the reader some idea as to
the real costs that are programmed by the DoD. The DoD receives the largest
budget of all the departments, year by year. The readers can see the year by
year increases in budget.
 5 • Life Cycle Costing 49

FISCAL TOTAL BASE OCO HURRI- PROCURE- RDT&E

YEARS DoD FUNDING CANES MENT
FY 2017 626.2B 543.8B 82.5B N/A 124.3B 74.1B
FY 2018 694.5B 623.4B 65.2B 5.8B 147.5B 92.0B
FY 2019 712.8B 641.1B 71.6B 2.8B 146.8B 95.5B
FY 2020 720B 642.2B 71.3B 8.0B N/A N/A
FY 2021 716.2B 647.2B 69.0B N/A 137.2B 106.7B

Additional budget data for FY 2021 is provided with a different break-

down, as follows:
NAVY ARMY AIR FORCE
Total-$ Total-$ Total-$
21.5B 12.8B N/A
Operations and Maintenance (O&M)
Navy Army Air Force
70.0B 73.0B 65.9B
Military Personnel (MILPERS)
Navy Army Air Force
55.2B 65.5B 42.8B
Military Construction (MILCON)
Navy Army Air Force
2.2B 1.0B 1.2B

REFERENCES
1. DAU (Defense Acquisition University), see dau.edu/cop/ce/dau.
2. Eisner, H., Computer-Aided Systems Engineering, Prentice Hall, 1988.
3. See www.fi-aeroweb.com/DefenseSpending.
4. Fiorello, Marco, Getting Real Data for Life Cycle Costing, RAND Corporation,
Santa Monica, CA, 1975.
5. See www.gsa.gov/node.
6. DoD Instruction 555.73, “Cost Analysis Guidance and Procedures”, Department
of Defense, March 13, 2020.
Cost Estimating
Relationships 6
There are formal procedures for estimating various types of system costs; these
are known as cost estimating relationships (CERs). This chapter provides an
overview and examples of CERs. These relationships provide answers for the
cost analyst using cost-effectiveness methods. Thus, the body of knowledge
represented by CERs is invaluable with respect to the overall topic of this book.
Some top-level CERs are presented in following table from an earlier
book from this author [1]:

TYPE OF EQUIPMENT COST ESTIMATING RELATIONSHIP (CER)

(DOLLARS)
Analog LOS Radio Equipment 6578 exp (.0001(3.94N + 208F))
where N = Number of voice grade channels, and F = frequency in Ghz

Parabolic Dish Antenna 65.9D2 + 2317

where D = diameter of dish in feet

Coaxial Cable 15.75L

where L = length of cable in feet

Tower-Mounted Passive Reflector 20.54A0.835 + 419

where A = area in square feet

Communications Satellite T, T and C 145 + 16.3W

where W = weight in pounds and T, T & C is telemetry, tracking and command

Super High Frequency Earth Terminal 0.085D +0.157P + 0.679 ($M)

where D = antenna diameter in feet and P = transmitter power (KW)

Communications Satellite Power 166P0. 501

($K)
Supply =
where P = beginning of life power (watts)

Comm. Sat. Attitude Control = 434 + 97.8W0.9 ($K)

where W = weight in pounds

DOI: 10.1201/9781003196174-6 51
52 Cost-Effectiveness Analysis

ANOTHER ILLUSTRATIVE LIFE CYCLE

COST – LINE OF SIGHT MICROWAVE
COMMUNICATIONS

A 12-year-life cycle cost estimate is as follows:

COST ELEMENT FY 1 FY2 FY3-12 TOTAL COST

($) ($) ($) ($)
R, D, T, & E Investment 0 0 0 0
Procurement
Microwave Equipment 111.8 166.0 — 277.8
Multiplex 150.2 219.0 — 369.2
Tech Control + 192.4 247.6 — 440.0
Order Wire 21.4 21.4 — 42.8
Alarm System 7.4 6.5 — 13.9
Electric Power 93.9 93.9 — 187.8
Integration/Assembly 28.8 37.8 — 66.6
Training 17.5 30.0 — 47.5
Test Equipment 86.5 113.3 — 199.8
System T&E 28.8 37.8 — 66.6
System Engineering 231.4 176.8 — 408.2
Project Management 75.5 57.7 — 133.2
Data 119.5 156.4 — 275.9
Contractor Tech Sup 53.4 39.8 — 93.2
Assembly, Install 230.6 302.0 — 532.6
Initial Spares 389.2 509.2 — 898.4
Transportation Construction 102.0 133.5 —
Site Activation 232.0 172.3 —
Annual Operating — — —
Military Personnel — — 308.0 3080.0
O&M — — 232.8 2328.0
Recurring Investment — — 107.2 1072.0
Operating Support — — 154.9 1549.0
TOTAL (Inc 10-Year 2172.3 2521.0 8029.0 12722.3
System Cost)
 6 • Cost Estimating Relationships 53

COCOMO
A critically important part of most systems these days is the software. Some
years ago, Barry Boehm took on the task of developing cost estimating re-
lationships for software. The result was two books [2,3] that can and should
be used by the systems engineer to formulate his or her LCC model for the
system under consideration.
For COCOMO I [2], the starting point is an estimate of the number of lines
of source code, as per a procedural language. From there, one is given a formula
that leads to the estimate of person months for the effort, given the mode of the
software development. For a mode known as organic in which we have small
teams, extensive experience and an in house stable environment, the formula is:

PM = 2.4(KDSI)1.05

where KDSI represents thousands of source instruction and PM is an estimate

of person-months. As an example, if KDSI is equal to 35,000, the person-
months are:

PM = 2.4(35)1.05 = 100.3 person months

For this mode, the formula for Development Time (TDEV) is:

TDEV = 2.5(PM)0.38 which becomes = 2.5(100.3)0.38 = 14.4 months

From this point we can easily calculate the Productivity, P, which is:

P = KDSI/TDEV = 35, 000/14.4 = 349 DSI/TDEV

The full-time equivalent staff then becomes

FTES = PM/TDEV = 100.3/14.4 = 7 FTES

Use of COCOMO I brings us to a simple calculation of four variables from

the input estimate of one of these variables, namely, KDSI.
For COCOMO II, the procedure is more complex and requires more
input parameters [3]. The basic form of the KDSI formula is the same, with a
leading coefficient A and an exponent B, as below:
Effort(PM) = A(KDSI)B
where PM = person-months, A is characterized by a set of effort multipliers,
and B represents a set of scale factors. A is itself a function of seven or
seventeen cost drivers, depending upon COCOMO I or II, respectively. The
54 Cost-Effectiveness Analysis

reference enumerates the scale factors for COCOMO II. So we see that an
excellent approach for the cost-effectiveness analyst is to use Boehm’s work
when trying to get good estimates of software cost. This is classical – using
the work of specialized researchers to assist in the processes of cost-
effectiveness analysis. Other approaches for cost estimation of software (such
as Function Points and Revic) are available in the literature [4].

LINEAR REGRESSION FOR CER

DEVELOPMENT [4]
The simplest form of CER is the linear one that is based upon a linear fit to a
set of data. The above cited COCOMO models are more complex than that;
the linear form is illustrated later.
We will suppose that we start with a set of data in (x, y) notation, as follows:

X (COST DRIVER) Y (COST)

4 30
3 60
7 90
9 70
10 100
10 120
11 140
14 140
15 170
17 140

We next take the line to be Y = mX + b.

And use the following relationships to solve for the slope, m, and the y-
intercept, b.

m = n xy x y/n xSquared ( s)squared and

b = sumysumxsquared sum x sum x y/nsum xsquared
(sum x)squared
leading to the following intermediate calculations:

sum x = 100 sum y = 1060 n = 10 sum xy = 12, 190 sum x squared = 1186
 6 • Cost Estimating Relationships 55

where by m = 8.5 and b = 20.5 and y = 8.5 x + 20.5

Going beyond the linear form we see the following non-linear forms:

Y = a(0) X a(1) Geometric or log-linear

Y = a(0) a(1) x Exponential
Y = 1/a(0) + a(1) X Hyperbolic

JOINT AGENCY CER DEVELOPMENT [5]

This handbook was sponsored by four agencies, namely:

1. The Naval Center for Cost Analysis

2. The Deputy Assistant Secretary of the Army
3. NASA, and
4. The Missile Defense Agency

It provides a variety of statistical approaches to equip the cost analyst in terms

of addressing a variety of estimating problems and data issues. It suggests a
six-step process for CER development:

1. Identify the purpose, scope, collection, validation, and normalizing

of data
2. Analyze normalized data
3. Generate the CER
4. Validate the CER
5. Characterize uncertainty
6. Document the CER

For this author, the matter of dealing with uncertainty is of special sig-
nificance. The analyst, as well as the user, wishes to know how much un-
certainty there is in the various cost estimates.
The regression methods for generating the CER itself include [6]:

a. Ordinary least squares

b. Generalized least squares
c. Transformational linear and log-linear model
d. Generalized linear model
56 Cost-Effectiveness Analysis

e. Non-linear least squares

f. Ridge regression
g. Minimum unbiased percentage error
h. Advanced regression methods

This source from the International Society of Parametric Analysts is a definitive

document from the folks who know most about parametric analysis. If your job
is the formulation of CERs, this handbook needs to be on your bookshelf.

OPERATING AND SUPPORT COST

ESTIMATING GUIDE [7]
Another view of OSD cost elements is provided in the Operating and Support
(O&S) Estimating Guide. This guide defines six major categories of cost,
along with the next level in the hierarchy of costs:

1. Unit level manpower

1.1 Operations
1.2 Unit level maintenance
1.3 Other unit level
2. Unit operations
2.1 Operating material
2.2 Support services
2.3 Temporary duty
3. Maintenance
3.1 Organizational maintenance
3.2 Intermediate maintenance
3.3 Depot maintenance
4. Sustaining support
4.1 System specific training
4.2 Support equipment replacement
4.3 Sustaining engineering and program management
4.4 Other sustaining support
5. Continuing system improvements
5.1 Hardware modifications or modernization
5.2 Software maintenance and modifications
 6 • Cost Estimating Relationships 57

6. Indirect support
6.1 Installation support
6.2 Personnel support
6.3 General training and education

The aforementioned breakdown illustrates at least some of the difficulty of

CER development, namely, that the CERs require basic data for estimation,
and that data needs to be collected in each and every one of the categories
shown earlier.

MITRE’S COST ESTIMATION

RESOURCES [8]
MITRE, the well-known systems engineering FFRDC, has taken the time to
identify several cost-estimation resources under their should cost management
and agile cost estimation initiatives. Several of these resources are listed later.
We note the emphasis on “should cost.”

• DODI 5000.73 Cost Analysis Guidance and Procedures, OSD,

CAPE, March 2020
• Should Cost Management in Defense Acquisition, OSD/AT & L,
Memo, August 2013
• AFI-65508 Cost Analysis and Guidance and Procedures, U.S. Air
Force, June 2012
• Should Cost and Affordability Memo, OSD/A, T & L, August 2011
• Navy: Implementation of Should Cost Management, July 2011
• Air Force: Implementation of Will Cost and Should Cost
Management, June 2011
• Joint Memo on Savings Related to Should Cost, OSD/A, T & L,
April 2011

From this listing, we can assume that other FFRDCs have similar lists, de-
pending upon their customers and their needs. These FFRDCs include the
Aerospace Corporation, RAND, and Anser. The reader is urged to contact
these enterprises to determine what data they have and what resources they
pay special attention to. In the area of cost estimation, the cost analyst does
not wish to miss useful sources of real data.
58 Cost-Effectiveness Analysis

REFERENCES
1. Eisner, H., Computer-Aided Systems Engineering, Prentice Hall, 1988.
2. Boehm, B., Software Engineering Economics, Prentice Hall, 1981.
3. Boehm, B., Software Cost Estimation with COCOMO II, Prentice Hall, 2000.
4. Eisner, H., Essentials of Project and Systems Engineering Management, Third
Edition, John Wiley, 2008.
5. Joint Agency CER Development Handbook, February 9, 2018, see www.dau.edu.
6. International Society of Parametric Analysts, Parametric Estimation Handbook,
Fourth Edition, April 2008.
7. Operating and Support Cost Estimating Guide (DoD), December 2020.
8. See https://aida.mitre.org/refereces/costs.
Cost-Effectiveness
and Benefits 7
A marker for the early consideration of the effectiveness of systems was the
so-called WSEIAC report [1−3], as cited in Chapter 1. This early definition
was brought forth in the year 1965, and the definition was that effectiveness
was measured by (a) availability, (b) dependability, and (c) capability. Of the
three, only availability was well defined at that time. But, in any case, this
chapter is a minor discourse on MOEs, what they are, and in some cases,
where they came from.
We will reiterate MOEs for various types of systems in the following
discussion.

A. Communications Systems
• Signal-to-Noise Ratio
• Detection Probability
• False Alarm Probability
• Range
• Speed of Service
• Grade of Service
• Bit Error Rate (BER)
B. Transportation Systems
• Trip Time
• PAX Capacity
• Freight Capacity
• Frequency of Service
• Distance/Range
• Speed/Acceleration
• Braking Distance
• Connectivity
• Capacity-to-Demand Ratio

DOI: 10.1201/9781003196174-7 59
60 Cost-Effectiveness Analysis

C. Air Defense Systems

• Target Detection Probability
• Tracking Accuracy
• Number of Targets in Track Mode
• Absentee Ratio
• Target Kill Probability
• Percent of Targets Killed
• Flux Required to Kill Target
• Kill Probability
D. Generic Information Systems
• Response Time
• Size/Capacity
• Security
• Defense against Malware
E. Generic Military Systems (Note Operations Research)
• Number of Mission Sorties
• Number of Aircraft Losses
• Required Aircraft per Mission
F. Delivery Systems
• Time to Deliver
• Required Vehicles for Delivery
• Down time per Vehicle
• Delivery Vehicle Capacity (Cubic Feet)
G. Household Systems
• Size of Lawn-Mowing Machine
• Capacity of Washing machine
• Size of TV
• Home Computer Size
• Power of Drying Machine
• Capacity of Refrigerator
• Mean-Time-Between-Failure (MTBF)

As we consider the aforementioned MOEs, we also recognize that the

Evaluation Criteria for the analysis of alternative systems are also drawn from
these MOEs. Usually, they are quite limited in number and they reflect what
system stakeholders find important about the systems they are considering.
For example, here are eight evaluation criteria that might be used in a variety
of systems analysis cases:
 7 • Cost-Effectiveness and Benefits 61

• Performance
• Availability
• Reliability
• Sustainability
• Human Factors (Ease of Use)
• Performance Risk
• Schedule Risk
• Environmental Effects

As we consider these MOEs, we take them to be commensurable, they can be

added even though they are, in general, of different dimensions. Such is the
basic nature of MOEs.

A MILITARY COST-BENEFIT ANALYSIS [4]

Cost-benefit analysis (CBA) in the military tends to move back and forth
between investments and divestments. The former involve projects and
programs, and adjustments in policy changes. The latter involve such
items as base closings and asset sales. An over-riding consideration has
to do with applying CBA to improving national security. Often, this
connection is not obvious but is there, none-the-less. The military budget
can often be a quite large discretionary item, and that makes the budget a
tempting target. It is known that especially vulnerable targets are military
and civilian compensation and the purchase and operation of equipment,
facilities, and services.
A context for military CBA is the more-or-less constant evaluation of
projects and programs. Examples of each are (a) infrastructure, and (b)
weapons, (c) UAV, and (d) CyberDfense. So, in general, military CBA is
used to move budgets up and down in order to:

a. Shape national security strategy within a given set of policies

b. Establish acquisition policies
c. Inform critical investments in people, and requirements

In general, there is considerable flexibility in military CBAs that accrues from

the sheer size of the military. An example is the SDI program that came into
being under Reagan. That program promised to be large and sustaining, but
the military seemed to have little difficulty finding its budget by some rapid
and clever adjustments that were not difficult to make.
62 Cost-Effectiveness Analysis

The military interest in CBA grew substantially after WWII and one can point
to developments in operations research and systems analysis as reasons. Work in
these fields combined economics, statistics, and decision theory. The RAND
Corporation was central to much of this work (see chapter one) and a key player
was Charles Hitch at RAND. Hitch and Roland McKean published an important
text [5], which explored ways in which military CBA can be applied, namely:

a. Guiding defense policy, and

b. Guiding defense investments

This seminal book, among other things, addressed the matter of applying
CBA to defense decisions and the often difficult matter of measuring benefits.
It points to the notion of “allocative efficiency” as central to what economists
focus upon when they are doing CBA. Under this concept, resources are
deployed to their highest value use in order to maximize social welfare. The
authors also claim that this is related to “Pareto Efficiency” where an allo-
cation is “pareto-efficient if no alternative allocation can make at least one
person better off without making someone worse off.” The so-called bottom
line from this source appears to be to:

adopt all projects that have positive net benefits.

As suggested, we are left with the often ambiguous issue of monetizing

benefits. This can be difficult, but certainly not impossible.

An Army Cost-Benefit Analysis to Support

Army Enterprise Decision-Making [6]
The Army has tackled the “CBA Problem” by releasing a complete guide to
carrying out a CBA from their point of view. They suggest the following main
steps for a CBA:

Step One – Develop the problem statement, define the objectives and scope
Step Two – Formulate assumptions and identify constraints
Step Three – Document the current state (status quo)
Step Four – Defines alternatives and cost estimation
Step Five – Identify quantifiable and non-quantifiable benefits
Step Six – Define alternative selection criteria
Step Seven – Compare alternatives
Step Eight – Report results and recommendations
The Army summarizes their cost-benefit analysis process by listing the costs
and benefits alongside one another. Here’s what that comparison looks like:
 7 • Cost-Effectiveness and Benefits 63

COSTS BENEFITS
The Total of Quantifiable and Total of Quantifiable and Non-Quantifiable
Non-Quantifiable Costs Benefits
• Quantifiable costs include: • Quantifiable benefits, including:
• Salary and benefits • Cost savings and cost avoidances
• Procurement • Non-quantifiable benefits, including:
• Sustainment • Greater capability
• Other costs • Faster availability
• Non-quantifiable costs include: • Better quality
• Opportunity costs • Improved morale
• Externalities • Other?

Other subjects alluded to earlier include:

• An alternative comparison decision matrix

• A briefing template
• Methods of data collection
• The work breakdown structure (WBS)
• Strategy for finding, using, and organizing cost data
• Recognition of the fact that there may be both quantifiable and non-
quantifiable benefits

The Army Cost-Benefit Analysis Guide (3rd Edition) declares that such a CBA
provide a review of the costs, the projected benefits, and the trade-offs that
might b required to pay for the system in question. The net result of the CBA
should be a strong “value-proposition” regarding the costs and the benefits.

Army Cost Analysis Manual, Dept. of the Army,

U.S. Army Cost and Economic Analysis Center,
May 2002 [7]
The Army has its own special way of approaching cost analysis. First, by way
of introduction, they look at relationships with:

a. The defense acquisition process

b. The PPBS process and systems, and
c. The contractual process
64 Cost-Effectiveness Analysis

Next, and the centerpiece of their cost analyses, appears to be:

• The analytical approach
• Cost-estimating methods
• Using current, constant, and discounted dollars
• Cost-estimating data sources
• Cost-estimation of software
• Cost as an independent variable (CAIV)
• Risk and uncertainty analysis
• Sensitivity analysis
• Validation analysis
• Interface with environmental and hazardous material impact analysis
• Cost-estimating errors
• Inherited assets
• Residual or salvage value

These topics are supplemented with material system special topics such as:
• Unit cost estimation
• Cost analysis requirements description (CARD)
• Cost analyses for major systems
• Interfaces with integrated logistics support (ILS)
• Force-level analyses

It is indeed interesting that the Army has decided to address both risk and
uncertainties. Both of these factors suggest that costs can be variable and at
times difficult to measure directly. This author favors “bracketing the costs”
when and where there appears to be various levels of uncertainty. This is
often not followed since it is difficult to do and also that it leads to serious
questions in both a CEA and a CBA.

COST-ANALYSIS PERSPECTIVES FROM

THE DAU [8]
As indicated in Chapter 5, the Defense Acquisition University (DAU) ap-
proaches this subject by the relevant program office preparing an initial life
cycle cost estimate (LCCE). The table of contents (TOC) for a defense system
includes all the costs necessary to research, develop, acquire, own, operate,
and dispose of weapon and support a system. For weapon systems, as an
example, this estimate of an LCCE helps to assess the affordability of the
 7 • Cost-Effectiveness and Benefits 65

program in question. It also is a required input to a cost-benefit analysis

(CBA) within the DoD. The LCCE uses base year dollars for a selected base
year, moving from there to a current-year dollar estimate. These are both used
in terms of the president’s budget that is submitted to Congress.

The categories of cost in an LCCE are as follows:

a. Research and Development (R&D)
b. Investment
c. Operating and Support (O&S)
d. Disposal

An example of the distribution of costs for various systems is provided in

Table 7.1.

TABLE 7.1 Distribution of costs for various system types

SYSTEM TYPE R&D INVESTMENT Q&S / DISPOSAL
Space 18% 66% 16%
Fixed wing aircraft 20% 39% 41%
Rotary wing aircraft 15% 52% 33%
Missiles 27% 33% 39%
Electronics 22% 43% 35%
Ships 1% 31% 68%
Surface vehicles 9% 37% 54%
AIS 30% 70%
Source: DoD estimate from 1999 [ 8].

For purposes of comparing life cycle costs, we can look at the following
breakdown:

Development Costs
a. RD T & E
b. Systems engineering
c. Program management
d. Test and evaluation
e. Support items
f. Initial spares and repair parts
g. Facilities
66 Cost-Effectiveness Analysis

MILCON
a. Facilities

Procurement Costs
a. Flyaway costs – procurement (prime mission, SE/PM, support)

DISPOSAL COSTS
Development Costs include all WBS elements for designs and tests. Flyaway
costs have to do with prime mission equipment (e.g., tanks, aircraft, and so
on). Weapon system costs are funded completely from procurement appro-
priations. Acquisition Cost is a multi-appropriations cost item. Life cycle
costs, by definition, are all the costs of the system in question.

REFERENCES
1. Weapon System Effectiveness Industry Advisory Council (WSEIAC), 1965, see
https://apps.dtic.mil.
2. Cost-Effectiveness Analysis of the Air Force C-17 Program, see https://
media.defense.gov.
3. The C-17 Program, L. Greer, Cost and Operational Effectiveness of C-17, IDA,
Institute for Defense Analysis, IDA Report Number R-390, 2010.
4. F. Melese, et al., Military Cost-Benefit Analysis: Theory and Practice, CRC
Press, 2015.
5. Hitch, C. and R. N. McKean, The Economics of Defense in the Nuclear Age,
Harvard University Press, 1960.
6. Cost-Benefit Analysis to Support Army Enterprise Decision Making, December
30, 2009, see also U.S. Army Cost-Benefit Analysis Guide, Third Edition, April
24, 2013.
7. Cost Analysis Manual, U.S. Army, March 30, 2020.
8. Introduction to Cost Analysis, Teaching Note, DAU (Defense Acquisition
University), February 2011.
Everyday
Cost-Effectiveness 8
This chapter explores everyday cost-effectiveness considerations. By ev-
eryday we mean that we are likely to run into these domains just about every
day. Our first example is from the information technology (IT) world. In
particular, we look at the laptop computer component of an IT system. We
explore a case where we move from a low-cost alternative, to a knee-of-the-
curve system, to a high-effectiveness computer element. We will look at such
factors as effectiveness as well as affordability as we make a choice as to
which system to invest in. The evaluation criteria define the perspective of
what various stakeholders might examine.

FUNCTIONAL DECOMPOSITION [1]

A computer system, in general, has the following functional decompositions:

• Input
• Output
• Database Management
• Cyber Protection
• Storage
• Processing
• Video Processing
• Operating System
• Applications
• Power Supply
We next look at a particular computer system, the small desktop.
Following the Eisner Architecting Method (EAM) [1] (and the AoA
Model), we construct three alternatives, as shown below:

DOI: 10.1201/9781003196174-8 67
68 Cost-Effectiveness Analysis

FUNCTIONS / SYSTEM A SYSTEM B SYSTEM C

SUBFUNCTIONS
1. Input Keyboard Keyboard+Screen Keyboard
+Screen+Disc
2. Output Screen, Printer, Screen, Printer, Disc, B with Higher
14“ Monitor 15“ Monitor Resolution 17”
Monitor
3. Database Mgmt Access IBM product (DB2) Oracle
4. CyberProtect Bitdefender McAfee High End Norton
5. Storage 512 Gb 16 Gb 16 Gb
6. Processor 2.3 GHz 2.4 GHz 2.6 GHz

This may be compared with the specifics of a set of laptops, as offered by

a local computer store [2], as follows:

SYSTEM ALPHA SYSTEM BETA SYSTEM DELTA

Lenovo Flex 5 14” Acer Nitro 5 15.6” MSI GL65 Leopard 15.6”
2 in 1 Laptop, AMD Laptop, Intel Intel Core i7 10th
Core i5 Generation
Ryzen 5 4500U 2.4 GHz Processor 2.6 GHz, Super 8 Gb,
16 Gb
Processor 2.3 GHz 16Gb RAM DDR4 RAM, 512 Gb
8 GB Onboard RAM 512 Gb Solid State HDD
512GB Solid State Drive
Drive
AMD Radeon
Graphics
PRICE [ 2]: $600 $1,000 $1400

We will evaluate the systems in question according to the following il-

lustrative evaluation and scoring system:

EVALUATION WEIGHT SYSTEM A SYSTEM B SYSTEM C

CRITERIA R WxR R WxR R WxR
Speed .3 7 2.1 8 2.4 9 2.7
Reliability .3 8 2.4 8 2.4 8 2.4
Cyber Protect .2 9 1.8 9 1.8 9 1.8
Capacity .2 7 1.4 8 1.6 9 1.8
TOTALS 7.7 8.2 8.7
 8 • Everyday Cost-Effectiveness 69

Beyond this illustrative case, we might include more evaluation criteria

and sub-criteria.

COST-EFFECTIVENESS OF HOME
PURCHASE/BUILD
We further illustrate an everyday use of cost-effectiveness analysis by ex-
ploring the building and purchasing of a new home [1]. The following listing
shows the overall functions and the approach to satisfying each function.

FUNCTIONS LOW COST KNEE-OF-CURVE HIGH

FOR HOME EFFECTIVENESS
1. Overall Style Ranch Faux Farm Contemporary
2. Home Single Furnace/AC 2 Zone Furnace/AC 3 Zone Furnace/AC
Environment Low Capacity Medium Capacity High Capacity
3. Food Standard Kitchen Extra Built-Ins Island + Extra
Preparation Counters
4. Sleeping Three Bedrooms Four Bedrooms Five Bedroom +
Facilities Den
5. Bathing Standard Add Sinks + Bidet Add Cabinets &
Toilet/Bath Jet Bath
6. Recreation Small Deck Add Small Rec Large Rec Room
Room Larger Deck Wrap around Deck
Add Brick BBQ Add Warmers
7. Space/Size 3,500 sq ft 4,500 sq ft 6,000 sq ft
8. Living/Dining Standard Spaces Larger Spaces High Ceilings
Space
9. Lawn/Garden Small Lawn Buried Water Lines Large Lawn &
Gazebo
10. Security No Extra Security Camera&Tape Add Alarms &
Internet
11. Auto Facility One Car Garage Two Car Garage Three Car Garage
12. Electrical Standard Amps Add 50% Add 100% for
Growth
13. Plumbing Standard Copper Multiple Flareouts Add Sprinkler
System

(Continued)
70 Cost-Effectiveness Analysis

FUNCTIONS LOW COST KNEE-OF-CURVE HIGH

FOR HOME EFFECTIVENESS
14. Built-Ins Standard Add Displays Into Living & Dining
Areas
15. Special Standard Closets Extra Room for Video or Game
Amenities Office Room
Add Built-In Add Elevator
Bookcases
COSTS $800,000 $1,500,000 $2,800,000

The illustrative evaluation criteria, weights, as well as ratings, are shown

as follows:

WEIGHT LOW COST KNEE-OF- HIGH

CURVE EFFECTIVENESS
RxW RxW RxW
Maintainability .3 8 (.3) = 2.4 7.5 (.3) = 2.25 6.5 (.3) = 1.95
Holds Value .2 7 (.2) = 1.4 7.5 (.2) = 1.5 8.5 (.2) = 1.7
Feels .1 7 (.1) =.7 8 (.1) = .8 9 (.1) = .9
Like Home
Overall Risk .2 8 (.2) = 1.6 8.5 (.2) = 1.7 9 (.2) = 1.8
Ease of .1 8 (.1) = .8 8 (.1) = .8 8 (.1) = .8
Upgrade
Neighborhood .1 7 (.1) = .7 8 (.1) = .8 9 (.1) = .9
SUMS 7.6 7.85 8.05

Looking at effectiveness to cost ratios, we obtain the numbers:

• House A = 7.6/800k = 9.5 (10)

• House B = 7.85/1500k = 5.23 (10)
• House C = 8.05/2800k = 2.88 (10)

We note that on the basis of these ratios as well as cost-effectiveness, for

House C we would be paying a lot of money for the increase in numerical
effectiveness. However, if one has the funds, the extra features seem to be
well worth the money. Or, perhaps House B looks just as good or even better,
considering how much less funding is necessary. Typically, these extra
 8 • Everyday Cost-Effectiveness 71

funding requirements show up as a monthly mortgage charge which hits the

monthly budget and at times would force one to cut back on other “neces-
sities” (like the baby needs a new pair of shoes).

INCREMENTAL CAPABILITIES AND COST

The normal procedure in evaluating alternatives is to do so with a common
and fixed set of functions. In other words, we do not evaluate one system
which carries out one set of functions, and a competing systems which has
one or more additional functions that it executes. However, the reader might
consider an evaluation process in which the capabilities for the different
systems are different. An example of such a procedure is cited later.
Suppose we are interested in purchasing a CD player. We do a search and
find that we can purchase a standard portable CD player for $27. We continue
to search and find that a CD player that also has an AM/FM radio. The latter
is a new function, and that configuration cost $32. We continue our search and
find the set of possibilities summarized below:

COST
• Original CD player $27
• CD plus AM/FM radio $32
• CD player plus radio plus cassette player $36
• CD/radio/cassette plus 5 CD changer $40
• CD/radio/cassette/changer plus remote $42
• CD radio/cassette/changer/remote plus iPod interface $45

This example shows increases in functions and their costs. Which

configuration would you be inclined to purchase? Is there a configuration
that you prefer that is not listed (e.g., a CD player plus radio plus 5CD
changer, with no cassette recorder). So the choice from among a set of
alternatives can be examined from the point of view of increased cap-
abilities and their costs. This notion can be applied to the purchase of an
automobile, as presented later.
We are interested in purchasing a new car but are not sure as to the
functions we wish to have. That choice depends in the main as to what the costs
will be for the different functions. We do some research and find the results
72 Cost-Effectiveness Analysis

listed later. Which choice will you make? The answer is likely based upon the
specific features of each, need, and affordability.

COST MPG
($) CITY / HIGHWAY
Honda HR-V 21,220 28/34
Honda CR-V 25,350 28/34
Honda CR-V Hybrid 30,560 40/35
Honda Pilot 32,550 20/27
Honda Passport 32,790 20/25

REFERENCES
1. Eisner, H., Systems Architecting, CRC Press, 2019.
2. See Micro Center website: Laptop computers.
Selected
Military
Examples
9
There is often considerable difficulty with respect to the cost-benefit analysis
of military systems [1]. So it’s not as if these are open and shut cases. Indeed,
back in 1984, the DoD implemented an initiative that called for the cost-
effectiveness of contract requirements. There is a considerable literature re-
garding specific cost and benefit analyses from the military. This chapter
provides information of this nature for cargo aircraft and also for the Joint
Strike Fighter, the latter being one of the more controversial cases and si-
tuations. Then, we take a brief look at some derivative programs from the SDI
program, the National Missile Defense program. This is followed by some
observations regarding the space command and cost growth.

C-17 EXAMPLE [2]

The C-17 Globemaster program is a significant success story and is, therefore,
well worth exploring in terms of its journey within the Department of
Defense (DoD).
This aircraft carries both cargo and personnel almost anywhere in the
world to include some remote airfields. It is flown by the Air Force’s Air
Mobility Command and several international air commands.
Notwithstanding these positive attributes, the program came close to
being completely cancelled in the early 1990s. Its story is also an indicator of
the value of cost-effectiveness analysis within the DoD. Its cost-effectiveness
profiles, if you will, illustrate the role of cost-effectiveness analysis in the
right type of environment.

DOI: 10.1201/9781003196174-9 73
74 Cost-Effectiveness Analysis

As part of the background for this cargo program, in December 1979, the
Air Force began a competition for the Cargo-Experimental (C-X) aircraft. The
requirements for this program included delivery of a full range of combat
equipment over intercontinental distances, operation from a 3000-foot airway,
survivability features, excellent RMA characteristics, and a low life cycle
cost. It was considered to be a multi-mission aircraft. An RFP was issued and
McDonnell Douglas won the competition.
Issues that complicated the procurement included an incorrect early
assessment of development and cost risk. Also, problems with reliability
and performance appeared. Lockheed Martin also came upon the scene with
its C-141 as a competitor. The key questions for the DoD, at that time,
appeared to be [3]:

Should it continue with the C-17 as designed in spite of escalating costs and
lower performance than originally thought?
Should it shift to the C-141 program by extending its life?
Should it explore yet other alternatives?

Such were these rather important matters facing the DoD in 1993 (among
others). The response was to request that a study be carried out which came to
the IDA to execute, which they did [3].
The IDA cost-effectiveness analysis was completed in 1994 leading ul-
timately to the conclusion and decision that the OSD approve a 120-aircraft
C-17 program in 1995. The latter type of analysis was critical to the ultimate
result, which was derived from:

a. The identification of fleet alternatives

b. The use of common capacity measures – million ton-miles per
day (MTMD)
c. The estimation of the effectiveness of each alternative

CARGO AIRCRAFT – THE NAVY

CARGO CASE
The scope of this cost-benefit analysis (CBA) had to do with replacing the C-9B
aircraft with the C-40A [4]. Three alternatives were considered, as follows:

1. Keeping the C-40A acquisition program on hold indefinitely

 9 • Selected Military Examples 75

2. Resuming the C-40A acquisition program as projected in

FY2015, and
3. Going back to the original C-40A program as envisioned in the
Naval Air Plan of 2030

These three alternatives were explored largely on the basis of operational

costs per flight hour as well as total costs over the full life of the program. The
CBA was considered as important in regard to satisfying the Navy’s Unique
Fleet Essential Aircraft mission.
The bottom line was the selection of alternative three above, that is, going
back to the original C-40A program plan as per the Naval Air Plan of 2030. The
essence of the decision was that alternative three mentioned earlier had the
lowest discounted costs as well as lower risk and better overall capabilities.

THE JOINT STRIKE FIGHTER (F-35, JSF)

The Joint Strike Fighter (JSF) program has been one of the most complicated
programs ever undertaken by the DoD [5]. Going back to 2007, we note the
analysis of costs of the JSF engine program. At that time, the JSF was cited as the
most expensive aircraft acquisition in the DoD. It was expected to develop, pro-
cure, and maintain 2443 aircraft at a cost of $338 billion over the program’s life
cycle. It had three variants within the program – a conventional take-off and
landing for the Air Force, a carrier-capable model for the Navy, and a short take-
off and vertical landing for the Marine Corps. The program also had very sig-
nificant promise for international sales. In 2007, the DoD reported on the remaining
costs under a sole-source scenario of some $54.3 billion for the engine program.
Benefits under this program were identified as better engine performance, relia-
bility, improved industrial base stability, and more responsive contractors.
Another data point is provided by the RAND Project Air Force team that
explored the costs and possible savings of joint tactical aviation acquisition
programs [6]. This report indicated that historical joint aircraft programs have
not saved overall LCC vs single service aircraft programs. Further, as of that
date (2013), the JSF was not on a path to savings LCC vs. a single service
approach. Not an encouraging picture, but still the JSF had enough forward
motion to continue.
If we move forward, closer to 2020, we find these significant events:

• Problems with software have arisen which could cause delays and
costly retrofits
76 Cost-Effectiveness Analysis

• This is bad news in terms of overall maintenance costs for the JSF
• The current goal for block four software is not achievable
• Unexpected software defects have been found after a quarter of the
software had been delivered by the prime contractor
• Others in Congress are looking at the overall program costs which
are nearly $2 trillion over its lifetime

On the positive side, international sales of the JSF are expected to be strong,
maintaining a revenue stream of some significance to Lockheed Martin. Japan
is expected to be the largest international customer with a purchase of 147
aircraft. The following countries are in formal partnerships with the United
States on this program:

• Australia, Canada, Denmark, Italy, the Netherlands, Norway,

Turkey, and the United Kingdom

Beyond these commitments, these countries have been cited as a part of the
Foeign Military Sales program, and very likely will be acquiring (or have
already acquired) the JSF:

• South Korea, Israel, Singapore, Greece, Romania, Spain, and Poland

In November 2020, Lockheed Martin was awarded a contract for what was
called low-rate initial production Lot 12. That indicated funding for a total of
255 aircraft – at a cost of $3.5 billion from the United States and $2.5 billion
from international countries. There are some 30–40 nations that have indicated
a willingness to purchase the JSF with the same price point as the F-16, namely,
$50 million per aircraft. Lockheed Martin has announced that it will sell the
F-35A for less than $80 million per plane, lower than the price of $89 million,
the Lot 11 deal. There is evidence of a willingness to purchase fewer but more
advanced (and costly) aircraft. Apparently, this is motion up the cost-
effectiveness curve, that is, more cost and more effectiveness as price points.
As a way of illustrating a cost-effectiveness analysis, we very briefly
examine a performance and cost analysis [7] of the A-10 Warthog vs. the F-
35 Joint Strike Fighter given the future need to deal with close air support. A
documented effectiveness analysis showed that the A-10 vastly outperforms
the JSF in providing close air support (CAS). Also, a cost analysis showed
that replacing the A-10 before the end of its service life (ending in 2035)
would cost at least $20.9 billion. The bottom from a cost-effectiveness point
of view – given the future need for close air support, we have less effec-
tiveness and more cost in terms of comparing these two alternatives.
 9 • Selected Military Examples 77

NATIONAL MISSILE DEFENSE [8]

It’s now March of 2021 and the new missile defense approach is on the
Deputy SecDev’s desk, awaiting approval. The new missile defense program
is designated as NGI, the Next Generation Interceptor. It’s been a long way
from the SDI program (of the Reagan era) to this interceptor, but here we are.
Two development contracts will be awarded as we move on down the road
toward a real defense against the enemy’s missile attacks. Will this solution
be effective? We still don’t know for sure, but the years of analysis and
experimentation suggest that the answer is “yes.” But systems can be over-
whelmed by increasing threat levels, so we must continue with what we think
is a cost-effective solution. We keep our eyes on the next couple of years of
work on this NGI and we check to see that the cost-effective solution that we
believe is ready for development. These decisions and actions are more than
paperwork; they have to do with ultimate survivability.
In 2020, the Missile Defense Program was described [9] in terms of some
five elements, namely:

1. The Aegis Ballistic Missile Defense

2. The Terminal High Altitude Area Defense (THAAD)
3. The Space-Based Infrared System (SBIR-High)
4. The Patriot Advanced Capability (PAC-#)
5. The Ground-Based Midcourse Defense

The Missile Defense Agency (MDA) had appropriated more than $200 billion
for the interval 1985 to 2019. U.S. policy with respect to missile defense had
as a goal to protect the homeland against limited long-range missile strikes
from countries such as North Korea and Iran. The focus has not been pro-
tection against the stronger states such as Russia and China. Defining the
“threat” in this way helps to define the “effectiveness” required for the sys-
tems that are part of our funded programs. Aside from the point-defense
Patriot missile system, none of the systems in the U.S. arsenal have in used in
combat. Various voices in the government and the press have argued that this
current status of missile defense is not sustainable. This argument basically
says that new defenses need to be added so as to reduce the cost of missile
defense and also be able to deal with possible increasing threats and levels of
threat. The latter include such threats as regional cruise and hypersonic
missiles. One answer to new and advance threats has been the Space-Based
Laser System which was funded at a level of $10.3 billion in 2019.
78 Cost-Effectiveness Analysis

A brief note regarding the costs and benefits of a cruise missile defense led
to the conclusion that the costs may well top the benefits. This was reported by
Tom Karako, Director of the Missile Defense Project at the Center for Strategic
and International Studies (CSIS). He declared that “we are not going to be able
to defend every acre of North America.” The numbers he quoted are: The costs
are in the range of $75 to $465 billion over ten years. Mr. Karako also pointed
out that he believes that the great differentiator for the program is its compe-
tition. He used the phrase “fly before you buy.” He also suggested that the
relatively longer development time for the NGI can be mitigated by near-term
improvements in ground missile defense. Looking at the matter from an ef-
fectiveness point of view, one might say that Karako is prioritizing capability
over capacity and stresses a conservative acquisition approach.
As another example, a 2021 report indicates that upgrades to the Iron
Dome Program are effective in defeating certain types of attacks from rockets
and drones. The question might be put: what are the most cost-effective
upgrades that would be candidates for change, and over what time frames, and
what would they cost? Thus, the budget changes are viewed in a context,
namely with respect to specific alternatives. Israel’s Raphael reports that the
Iron Dome is the world’s most deployed system against rockets, mortars,
artillery shells, and air breathers. They report a success rate of more than 2500
intercepts and also 90%.
The measures of effectiveness for the U.S. NGI program (Next
Generation Interceptor) is a Ground-Based Midcourse Defense Interceptor are
advertised as affordability, transparency, rapid development, lean manu-
facturing, and low-risk accelerated schedule. The Ground-Based Missile
Interceptor consists of a multi-stag booster and an Exo-Atmospheric Kill
Vehicle (EKV). The EKV separates from the booster in space and looks for
targets via radar and onboard visual and infrared sensors. The EKV may also
be called a kinetic-kill vehicle in that it destroys its target by colliding with it.
Other terms that have been sed in this regard are hit-to-kill and kinetic kill.

EFFECTIVENESS IN SPACE [10]

We introduce here the notion of effectiveness measurement as a management
perspective. The question of interest might well be:

How effective is this new command known as the space command, or any
other command for that matter?
 9 • Selected Military Examples 79

Applying the concept of an MOE, we can explore the effectiveness of a unit that
is part of any organization. We first ask: “What is the mission of that unit?” and
then go on to: “What activities are being carried out to fulfill that mission?” and
then on to: “How well is the unit executing those activities?” This concept is
analogous to Drucker’s Management by Objectives (MBO idea). Thus, orga-
nizational behavior is quantified and evaluated from a management point
of view.
In the case of the new (reestablished in 2019) space command, the
mission is stated as [11]:

to conduct operations in, from, and through space to deter conflict, and if
necessary, defeat aggression, deliver space combat power for the joint/
combined force, and defense U.S. vital interests with allies and partners.

It, thus, appears that the space command has wide latitude for conducting
global space operations when such actions appear to be called for. The reader
will recall that the Command contains the well-known and advertised Missile
Warning Center in Cheyenne Mountain, Colorado. If we look at the mater of
MOEs for that facility, we quickly are able to identify the following:

a. Response time
b. Information friend or foe
c. Probability of detection
d. Number of false alarms
e. State(s) of readiness

The Command also has a major space operations center Vandenberg Air
Force Base, California.

CUTTING THE DEFENSE BUDGET [12]

From time to time, we see “cutting the defense budget” as a partial solution to
the many problems that lay ahead for this country. Depending upon how it’s
done, this is precisely the wrong idea to be explored in the context of cost-
effectiveness or cost-benefit analysis. The meta-cleaver approach does make
new monies available, but for what? It lacks specificity and also program
details. What programs leave, and what programs are new? What is compared
with what, and what are the alternatives?
80 Cost-Effectiveness Analysis

Discussions regarding cutting the defense budget were front and center in the
year 2021 as the new administration started to look at the overall budget such that
the military budget would be slashed from $741 billion the prior year. A 10%
reduction in defense spending was contemplated, with an increase in non-defense
spending (health, education, foreign aid, housing, and so on). As suggested, this
type of budget manipulation is not generally part of true cost-effectiveness analysis,
which tends to be true to comparative analysis of alternatives.
As we close this chapter, we take note of the fact that some analysts think
that a smaller budget will result in a more effective military [13]. This source
points to a reduction in service contracting off 10% to 15%, which would save
some $175 billion to $262.5 billion over a period of ten years. Exploring the
“new” independent space force, they come to the conclusion that by not
creating a separate service, we could save some $13 billion over the next five
years. They also claim that a larger overall defense budget results in a less
capable force. Program-specific points that they make with respect to the DoD
budget are the following:

a. The cost of the F-35 program grew by as much as 89% above the
original baseline
b. The cost for the littoral combat ship more than doubled from the
original $220 million estimate
c. The cost of the Army’s future combat systems program experi-
enced a 76% growth before it was abandoned
d. The cost of the Zumwalt-Class Destroyer grew by 45.1% before the
Navy stopped production after launching 3 of the planned 32 ships
e. The Ford-Class Aircraft Carrier Program costs grew an average of
21% per ship

A bottom line? It seems that a real truth is that we are not able to bring many
important programs within the estimated cost, as shown by real data. However,
we have not really proven that a smaller budget leads to a more effective
military. Further, cost growth in military systems is real, and is also reinforced
by the literature [14]. RAND identifies four sources of cost growth, namely:

a. Errors in estimation and scheduling

b. Decisions made by the government
c. Financial matters, and
d. Miscellaneous sources

A GAO Report [15] claims that Major Defense Acquisition Programs

(MDAPs) have grown in cost by an estimated $296 billion. Selected acqui-
sition programs that were examined include the following:
 9 • Selected Military Examples 81

SYSTEM DATE OF MS-II/B

APPROVAL
1. Armed Reconnaissance Helicopter July 2005
2. Expeditionary Fighting Vehicle Dec 2000
3. Future Combat System May 2003
4. Global Hawk Feb 2001
5. Joint Air-to-Surface Standoff Missile Nov 1998
6. Joint Strike Fighter Oct 2001
7. Joint Tactical Radio System–Ground Mobile June 2002
Radio
8. Littoral Combat Ship May 2004
9. Amphibious Transport Dock June 1996
10. Space-Based Infrared Satellite–High Oct 1996
11. Warfighter Information Network–Tactical July 2003

A NOTE ON TRANSPARENCY
We take note here in closing this chapter that the Biden Administration has
decided not to include a five-year spending plan for 2022. This allows precedent
from the prior three administrations. Hence, there is less visibility into specific
budget plans, and possibly a slowdown in the extent to which cost-effectiveness
analysis is carried out and shared with Congress and the industry. This means
less understanding of, and details regarding, such items as the JSF, big-deck
aircraft carriers, hypersonics missile programs, Army ground vehicle moder-
nization, and long-range precision weapons. Incidentally, the DoD budget is
$715 billion, a 1.56% increase over the previous year.

REFERENCES
1. Difficulties of Military CBA, see https://medium.com
2. Globemaster2, see https://apps.dtic.mil
3. C-17 IDA Study, An Applicaiton of Cost-Effectiveness Analysis in a Major
Defense Acquisition Program, IDA Report R-390, 2010.
82 Cost-Effectiveness Analysis

4. Navy Cargo Example, see https://navy.usni.org.

5. Joint Strike Fighter (JSF), Wikipedia, JSF program.
6. RAND Project, Joint Tactical Aircraft Programs, see https://www.rand.org
7. Cambridge Core, Thunder vs. Lightning, A Performance and Cost Analysis of
the A-10 Warthog vs the F-35 Joint Strike Fighter, Journal of Benefit-Cost
Analysis, Volume 10, Fall 2019.
8. National Cruise Missile Defense Report: Issues and Alternatives, see https://
cbo.gov.
9. MDA, Missile Defense Agency, see https://www.mda.mil.
10. Space Command, see https://www.spacecom.mil, 2021.
11. Space Effectiveness/Mission, Wikipedia.
12. Slashing Defense Budget, see https://www.defense news, 9/21/2021.
13. Smaller budgets, more effective, see military, www.pogo.org.
14. J. Bolten, et al, “Sources of Weapon System Cost Growth”, see www.RAND.org.
15. GAO Report, see https://www.gao.gov.
Miscellany
10
Two key points that have been raised in the main body of this book are:

a. It is important to clearly define measures of effectiveness for

alternative systems, and
b. Both the CEA suggested here and the AoA suggested elsewhere
settle on three alternatives as a useful and practical approach
to CEA

It is recommended that the analyst take these perspectives into account in

performing a CEA (or a CBA). The three alternatives approach is also totally
consistent with the systems engineering practice of defining the synthesis of a
system’s architecture.

RISK ANALYSIS
An article in the Washington Post [1] reported on the findings of the Centers
for Disease Control and the Food and Drug Administration by declaring that:

the officials said that the benefits of the single shot vaccine far outweigh the
risks from a rare and severe type of blood clot.

The background regarding this review is simply that a limited number of

blood clots followed the single-shot vaccine, and so the situation was more
carefully investigated. This brought the notions of benefits and risk into clear
focus, which is often the case with respect to medicines and various types of
foods. The question is – we try to balance benefits and risks to come to some
conclusion about whether or not it is safe to take the medicine or food.

DOI: 10.1201/9781003196174-10 83
84 Cost-Effectiveness Analysis

Usually, there is no all-purpose answer to that question, but it requires a deep

look at the particulars of each case and situation. With the Johnson & Johnson
one-shot vaccine, there were six cases of blood clots among the millions of
people that had received the vaccine. The small number of clots were enough
to convince the reviewers that the benefits were much greater than the risks,
although the risks were not zero. Again, this is a scenario that is likely to find
in the health field. It brings risk into the equation to be considered. From time
to time, this notion of risk is extended by looking at “cost risk” in any
situation.

BUSINESS CASE ASSESSMENT

If one looks at the curricula of various MBA programs, they will find at-
tention being paid to “business case assessment” or “business case analysis”
or just plain business cases. The Harvard MBA program is well-known for its
focus on business cases. Indeed, they form the basis for Harvard’s overall
approach to learning in the business arena. If we look more carefully at
business case assessments, we find that many MBA programs have this as a
central topic or course. The Wiki folks have cited several reasons for creating
a business case [2].

a. They help decision makers select a high value initiative from

among a set of alternatives, and
b. They provide follow-up specifics about what to do to realize the
preferred alternative.

Some of the key elements of the business case “report” are cited as [2]:

1. Summary and recommendation

2. Key assumptions
3. Costs
4. Benefits
5. Cash flow projections
6. Risk
7. Strategic options
8. Opportunity costs
9. Immediate first steps
 10 • Miscellany 85

We especially note that the aforementioned points contain costs and benefits,
but not effectiveness measures.
The MBA program at Harvard, for example, is known as an excellent
way to convey ideas about how businesses are run. The Harvard “CASE
STUDY” approach has been well-documented, as well as accepted. For ex-
ample, the Harvard Business Review has provided a detailed analysis of what
Jack Welch did at GE [3].
In 2021, we see an example of a business case analysis in the military
arena [2]. The Defense News reported that the DoD was doing a business case
assessment relative to organic maintenance on the F-35 aircraft This started
out by defining elements of the case and considerable bartering with
Lockheed Martin, prime contractor for the F-35. Areas for exploration in-
cluded the increasing operating costs, chronic spare parts shortages, and what
they called deeply troubled maintenance software and block 4 software de-
lays. The business case assessment involved consideration of the myriad af-
fecting factors and concluding how much maintenance should be done by
Lockheed Martin and how much should be shifted over to more organic
maintenance performed by the DoD services. This addressed the long-term
needs and the best approach that would be part of a new contract with
Lockheed Martin. So the business case assessment looked at the most “cost-
effective” alternative to the maintenance issue on this very important and very
expensive program. So, the “business case assessment” is considered to be
alive and well in the DoD. It is also a good expression to use when the DoD is
trying to explain their choices to Congress.

BUDGETING
A budget, of course, is a plan for expenditures. Budgeted items are not ne-
cessarily based upon CERs, but aspects of a budget might be. In many cases,
budgets are statements of what can be afforded, just to make sure that one
does not spend more money than what has been allocated.
When the real world intercedes, as it often does, costs are actually in-
curred and then, but only then, in many cases, is the question asked:

Are we within budget, or not?

If we are, then all is well. If we’re not, there’s much gnashing of teeth and the
“fixers” go into action.
86 Cost-Effectiveness Analysis

Budgets can be rather static, but when one looks over time, we see
tendencies and trends. The same is true with costs, observing real costs over
one or more time periods. These tell a story, whether intentional or not. So,
we examine if the budgets are going up, down, or sideways to learn new
truths about an organization or an enterprise.
So we are looking at the defense budget for the year 2022 and we find
that software, missiles, and testing are declared to be at the top. The comment
is that it’s the software and the munitions that “make a critical difference in
our capability” [4]. These tell the story, says Deputy Secretary of Defense
Hicks, and are great indicators of what is important as we move into a future
defense posture. And as we do, we need to pay more and more attention to
actual costs and where they are within budget or not.

COST-EFFECTIVENESS AND WARFIGHTING

Using cost-effectiveness principles are likely to be with us for quite a while,
even when it come to warfighting. One might call this disposition the legacy
from Secretary Robert McNamara, starting in the 1960s, and continuing on into
today. The SecDev Office of Planning, Assessment, and Evaluation (P, A, & E)
is still trying to bring quantitative analysis methods to the difficult topic of
warfighting. Recently, we have had to confront the matter of:

Do we bring most of our troops home from Afghanistan?

Is this a cost-effective position, or is it not? President Biden confirmed that

after 20 years it’s finally time to bring our troops home. Not even all our
senior military agree that this is the right course of action, but as of this
writing, we’re moving ahead in this important action.
Another important question is:

Do we continue to spend dollars to take care of cost growth, in the name of

defense, and when is this action crossed “the tipping point?”

And the answer is, as with Joint Strike Fighter, yes, according to our beset
minds that are able to consider a problem with many, many influencing
factors. That’s what the Joint Chiefs have on their plates. As Secretary Alain
Enthoven once said, when is enough enough?
 10 • Miscellany 87

WARFIGHTING
We know where Secretary McNamara came from and we understand his
perspectives from his writings such as The Fog of War [5]. Its critical to get
the data, he declared, using his austere and quantitative training at the likes of
Harvard and the Ford Motor Company. But there’s a perspective about
warfighting that comes from several generations earlier, namely, from Sun
Tzu [6]. Does he suggest anything like cost-effectiveness analysis? Not from
this author’s review, but the reader’s encouraged to dig more deeply into this
question.

FORT LEAVENWORTH KANSAS:

THE ARMY WAR COLLEGE
Fort Leavenworth Kansas is the location of the U.S. Army Command and
General Staff College. The College has four schools, namely:

a. Command and General Staff School

b. School of Advanced Military Studies
c. School for Command Preparation
d. School of Advanced Leadership and Tactics

This author noted that this “war college” offered a master of military art and
science degree, and wondered if preparation for dealing with the Office of the
Secretary of Defense was part of the curriculum. Specifically, did the War
College have “cost-effectiveness analysis” of systems as a field of study.
This led to the statement that military science deals with the “technical
dimensions of war and military operations.” And here is the operative
quote:

military art is generally subject to quantitative rather than qualitative in-

vestigation, although qualitative methodologies are used when appropriate.

This led this author to conclude that it is that cost-effectiveness analysis of

military systems is very likely to be one of the quantitative methods used.
88 Cost-Effectiveness Analysis

INTERCITY TRANSPORTATION EXAMPLE

Looking back at Chapter 2, we see an example of another perspective on
effectiveness and cost analysis. That perspective was first addressed in a book
by this author [7] in a chapter on performance analysis and modeling. It dealt
with alternative freight transportation systems, in particular systems with the
names as follows:

a. 40-foot motor carrier

b. TRAILS – a conceptual high-speed rail system defined as state-of-
the-art in high technology guideway operation
c. TOFC – a trailer or flat car dedicated 70-car train
d. NB Igloo – a narrow-body containerized air-freight system
e. WBLH – a wide-body lower-hold containerized air freight system

A selected number of effectiveness measures for these systems are listed as

follows:
Effectiveness Measures for Intercity Transportation Systems

• Passenger mobility index (PMI) – defined as the product of service

frequency and capacity, divided by the trip time
• Freight mobility index – same for the PMI, but applicable to freight
service
• Normalized PMI (NPMI) – defined as the PMI divided by the cost
per passenger-mile
• NPMI per unit cost
• NPMI per unit cost – normalized by the cost per ton-mile
• NPMI per energy usage – energy defined as the number of BTUs
per passenger-mile or ton-mile
• NPMI per energy/cost expenditure
• NPMI per pollutant/noise contribution – noise easured by popula-
tion affected within a particular noise exposure factor
• Viability index – defined as the net present value (NPV) divided by
the present value of R&D and investment cost

We see that even for a simple transportation example, there are quite a few
effectiveness measures that can be examined in an attempt to select the best
system from among a set of alternatives.
 10 • Miscellany 89

RATIONAL CHOICE THEORY [8]

The cost-effectiveness approach to selecting best systems is ultimately based
upon the decision-maker making what is known as a rational choice. There is a
formal theory that attempts to explain what a rational choice means. As sug-
gested in the literature, a perspective on rationality is instrumental, which that
involves achieving a goal using the most cost-effective procedure. This ap-
proach ties rationality to cost-effectiveness. When applying cost-effectiveness
principles, we normally assume that the analysts is rational and not biased in
any way. This can be difficult to achieve since the world, as we have come to
know it, accepts many forms of advocacy, which is known to be highly biased.

NASA: SYSTEMS ENGINEERING AND

COST-EFFECTIVENESS [9]
In their treatment of systems engineering [9], NASA has addressed the topic
of cost-effectiveness. Their top-level perspective is stated as:

the objective of systems engineering is to see that the system is designed,

built and can be operated so that it accomplishes its purpose safely in the
most cost-effective way possible, considering performance, cost, schedule
and risk.

This ties systems engineering to cost-effectiveness in quite a direct way. The

NASA approach is called the systems engineer’s dilemma, declaring that at
each cost-effective solution one must try to:

a. “reduce cost at constant risk, whereby performance must be

reduced
b. reduce risk at constant cost, whereby performance must be reduced
c. reduce cost at constant performance, whereby higher risks must be
accepted
d. reduce risk at constant performance, whereby higher costs must be
accepted”

This is an interesting perspective, not necessarily accepted by all. For example,

some years ago, Dan Goldin was the NASA Administrator and declared that he
90 Cost-Effectiveness Analysis

was going to try the “perfecta” – to improve all three, namely, schedule, cost,
and performance. Whatever the intention, the real world has somehow con-
spired to make it difficult to improve these three system features, all at the same
time. If we look at risk as well, we are likely to have to find trade-offs that work
in any particular real system.

MITRE’S AFFORDABILITY, EFFICIENCY,

AND EFFECTIVENESS [10]
MITRE is a well-respected think tank that has specialized in supporting the
Air Force for many years. They have covered a wide range of topics, with
systems engineering perhaps being the most prominent. One of their docu-
ments provides some insights into the topics of affordability, efficiency, and
effectiveness. [10]. Explanations from MITRE in this regard deal with:

• Affordability is the ability to fund a desired investment.

• Efficiency is the unit benefit per dollar.
• Effectiveness is the ability to achieve an organization’s mission.

MITRE provides the statement that solutions are effective if they deliver
capability of high value with respect to the mission(s) of the user community.
Looking more deeply into affordability, we note that a system may be
considered affordable if indeed it is within one’s budget for the system. In
contrast, for an expensive system (such as the Joint Strike Fighter), the sleight
of hand is to re-budget, or cut funds from one program to cover the expected
high costs of another. So a system that may not be inherently affordable all of
a sudden becomes affordable as the budget for that system is increased.

FROM THE MILITARY, AFFECTING

BUDGET [4]
• The Air Force continues to invest in the Joint Strike Fighter, de-
spite its high cost. It plans to buy 48 F-35A aircraft, along with 12
F-15EXs for $4.5 billion and $1.3 billion. The expectation is that
 10 • Miscellany 91

an increased buy will lower the per-aircraft cost. At the same time,
the Air Force is looking to retire some 201 planes.
• The Air Force is also focused upon better mining and sharing of
existing data. Several courses of action were considered during a
Data and Infrastructure Summit, in this regard. The Air Force is
supporting agile software development and innovation for the B-21
Bomber.
• The DoD was seeking an increase in the budget for the Cyber
Mission Force.
• The Biden budget cuts ships and planes, but calls for an increase in
Research and Development (R&D) for FY 2022. This is expected
to support advanced capabilities in hypersonic missiles, AI and 5G.
• The Army will select a single vendor for a cruise missile killer
capability. The focus is to counter cruise missiles and large drones.
• The United States is talking with Israel with respect to buying more
Iron Dome systems. This air defense system has demonstrated its
utility during the rocket shelling of Israel by Hamas. The United
States is extremely interested in this capability, given the current
assessment of threats. Israel also has offered to the United States a
long-range naval strike missile: Sea Breaker.
• The JAIC has been automating DoD’s back office, from minutes to
seconds.
• Air and space officials claim that the Digital Design Revolution is
the key to operations in all domains.
• NORTHCOM’s Gide experiments call for applying AI decision-
making tools to realistic scenarios to show what’s available now
and to facilitate culture change into the future.
• CNAS claims that the Biden FY22 budget and strategy calls for
trying to do too much with too little.
• Rear Adm. Bill Houston described a submarine that has the pay-
load and speed of the Seawolf-Class, the acoustics and sensors of
ethe Virginia-Class, and the operational availability and service life
of the Columbia-Class.
• Boeing is the prime contractor for the design, development, test and
production of the launch vehicle core stage, upper stages, and
avionics suite for NASA’s Space Launch System (SLS). This system
will allow the United States to explore the moon, Mars, and beyond.
• JADC2 (Joint All-Domain Command and Control) connects dis-
tributed sensors, shooters and data from all domains to all forces.
This provides secure networks and a decisive advantage to the
warfighter by connecting platforms and weapons via a common
architecture.
92 Cost-Effectiveness Analysis

• Georgetown’s Center for Security and Emerging Technology is-

sued a report that claimed that the DoD should stop being an
“innovation tourist.” It suggests a plan of action of three interesting
ways to proceed with innovation.
• Biden has issued a National Security Memorandum on critical in-
frastructure. The goal is to voluntarily adopt better cybersecurity
standards. What will that impact the budget. We shall soon find out.

Each of the aforementioned military notes refers to activities that may impact
the budget and ultimately require a cost-effectiveness analysis. Stay in touch
with “breaking defense” notes [4] to find out and track the actions taken by
the DoD and further references to cost-effectiveness analysis.

REFERENCES
1. “U.S. Lifts Pause on J&J Vaccine”, Lena Sun and Carolyn Johnson,
Washington Post, April 24, 2021.
2. Wikipedia, Business Case, see https://een.wikipedia.og/wiki/business_case.
3. Business Case, see https://www.hbs.edu.
4. See Breaking Defense.com.
5. The Fog of War, see en.wikipedia.org, see Film, Documentary, 2003, see also
book The Fog of War, by Blight J. and J. Lang, Rowman & Littlefield, 2005.
6. Sun Tzu, The Art of War, Shambhala, Boston & London, 2005.
7. Eisner, H., Computer-Aided Systems Engineering, Prentice Hall, 1988.
8. “Rational Choice”, see https://www.investopedia.com.
9. Systems Engineering Handbook, NASA, see www.nasa.gov/seh.
10. “Affordability, Efficiency, and Effectiveness” (AEE), MITRE, see www.mitre.org.
Summary
11
This final chapter provides a summary of the main points in the book, chapter
by chapter.
Chapter 1 looks back at the origins of the notion of effectiveness and
locates the Weapon System Effectiveness Advisory Committee (WSEIAC)
report. In that report, dated 1965, we find effectiveness defined as the pro-
duct of:

• Availability
• Dependability
• Capability

From there, we move on to a brief look at McNamara and the Whiz Kids. In
particular, the evolution to SecDef on the part of McNamara, and key posi-
tions by the other Whiz Kids, some at the Ford Motor Company and some in
government. Alongside of McNamara was Alain Enthoven, who explored
“how much is enough,” and set forth some six principles of his approach to
military systems planning and evaluations as Assistant Secretary of Defense
for Systems Analysis.
Finally, the RAND Corporation is introduced as an important force in
cost-effectiveness analyses. RAND started with solving military problems
and branched off into other issues and other agencies.
Chapter 2 provides an overview of the basics of cost-effectiveness ana-
lysis. First, there’s a citation of typical MOEs in transportation, commu-
nications, and air defense. Partial lists are reiterated as follows:

• MOEs for Transportation

• Passenger Capacity
• Freight capacity
• Storage capacity

DOI: 10.1201/9781003196174-11 93
94 Cost-Effectiveness Analysis

• Capacity-to-demand ratio
• Trip time

• MOEs for Communications

• Speed of Service
• Grade of service
• Detection probability
• False alarm probability
• Signal strength
• Noise power
• Signal-to-noise ratio

• MOEs for Air Defense Systems

• Target detection probability
• Number of missiles being tracked
• Absentee ratio
• Target kill probability

An evaluation framework for a typical transportation system is presented,

with a set of explicit evaluation criteria, namely (1) trip time (2) frequency of
service, (3) pollution, (4) noise, and (5) reliability. Effectiveness values are
calculated, and cost estimates complete the short form cost-effectiveness
assessment. Weighting factors are made explicit, drawn from a real-world
aviation advisory commission report.
The third chapter looks at cost-effectiveness and systems engineering,
especially on the relationship between cost-effectiveness and the systems
architecting process. The essence of the SEBoK is reiterated as:

a. SEBoK Introduction
b. Foundations of Systems Engineering
c. Systems Engineering and Management
d. Applications of Systems Engineering
e. Enabling Systems Engineering
f. Related Disciplines
g. Systems Engineering Implementation Examples
h. Emerging Knowledge

The key analysis step in architecting an IT system is briefly examined. It is

noted that both the architecting process and the AoA process both call for a
three-system alternatives. This is an interesting result of studying both the
 11 • Summary 95

subject areas. Finally, some attention is paid to MITRE’s Systems Engineering

Guide and the INCOSE Systems Engineering Handbook. The former em-
phasizes affordability, efficiency, and effectiveness. The latter is a compre-
hensive exploration of systems engineering, and is considered an activity that
requires updating from time to time.
Chapter 4 provides selected commentaries on cost-effectiveness. This
chapter returns to SecDef McNamara and his accomplishments. This is fol-
lowed by a history of cost-effectiveness analysis including a critique by E. S.
Quade of the RAND Corporation. The McNamara discussion includes:

a. Comments about Admiral Rickover

b. The PPBS (planning, programming, and budgeting system)
c. Examples of key programs

Selected courses are cited from the University of Maryland, Harvard, The
George Washington University, and the Pardee RAND Graduate School.
A bottom line is presented that suggests we are interested in three regions
of ensembles of systems:

a. The low-cost–low-effectiveness region

b. The knee-of-the-curve region
c. The high-effectiveness region

This applies to new systems that are being considered for funding, but also systems
that are available commercially. So if we are looking for a low-end computer
system, we might go to a computer store and look at the systems listed as follows:

a. Lenovo
b. Microsoft Surface
c. HP Pavilion
d. Apple iMac
e. Dell
f. Samsung Chromebook

Chapter 5 introduces life cycle costing as the cost element of cost-

effectiveness. Such costing is often called a “life cycle cost model” (LCCM)
and in DoD has the familiar categories as follows:

a. R, D, T, & E
b. Procurement
c. Operations and Maintenance (O&M)
96 Cost-Effectiveness Analysis

From DoD’s perspective, they often analyze budgets that also look at (a)
MILPERS (military personnel), (b) MILCON (military construction), and
overseas contingency operations (OCO). The DoD cost guidance is reflected
by the following key points:

• A report provided in a Cost Assessment and Program

Evaluation (CAPE)
• The articulation of methodology under the subjects:
A. Fleet alternatives
B. Airlift requirements
C. Effectiveness of alternatives
D. Cost estimates

Chapter 6 deals with the important topic of cost-estimating relationships

(CERs). Examples from the field of communications are cited, with con-
siderable detail devoted to line-of-sight microwave communications. This is
then transitioned into a discussion of Constructive Cost Model (COCOMO),
which is a leading edge software CER supported by years of research, data
gathering, and documentation in the form of two dense books. A feature of
COCOMO I is that one input estimate (KDSI – thousands of delivered source
instructions) leads to four distinct output estimates, namely:

1. Person-months (effort)
2. Development time (months)
3. Productivity
4. Full-time equivalent staff (FTES)

This is followed by a derivation of a linear cost model using sample data and
regression equations. Other forms of CERs are cited, to include:

a. Geometric
b. Log linear
c. Non-linear least squares
d. Ridge regression
e. Advanced regression

The chapter is concluded by a brief look at MITRE’s cost-estimation re-

sources, which are enumerated as:

• The DODI 5000.73, Cost Analysis Guidance and Procedures, OSD

• Should Cost Management in Defense Acquisition, OSD/A, T, and
L, August 2013
 11 • Summary 97

• Cost-Analysis Guidance and Procedures, U.S. Air Force

• Should Cost and Affordability Memo, OSD/A, T, and L,
August 2011
• Navy: Implementation of Should Cost Management, July 2011
• Air Force: Implementation of Will-Cost and Should Cost
Management, June 2011
• Joint Memo on Savings Related to Should Cost, OSD/A, T, and L,
April 2011

These “should cost” perspectives clearly had an impact on how the DoD
approached the overall topic of cost estimation.
Chapter 7 begins with an expansion of the sets of MOEs beyond the
original sets in Chapter 2, which were (a) communications systems, (B)
transportation systems, and (c) air defense systems. In particular, other types
of systems considered were:

a. Generic information systems

b. Military systems
c. Delivery systems
d. Household systems

For the sake of completeness, the following eight quite common evaluation
criteria were reiterated:

1. Performance
2. Availability
3. Reliability
4. Sustainability
5. Human factors (ease of use)
6. Performance risk
7. Schedule risk
8. Environmental effects (noise, pollution ...)

This is followed by a brief examination of military cost-benefit analysis

(CBA), with the bottom line being:

adopt all projects that have positive net benefits

And the main steps for a CBA (from the Army) being:

1. Develop the problem statement, including the objectives and scope

2. Formulate assumptions and identify constraints
98 Cost-Effectiveness Analysis

3. Document the current state

4. Define alternatives and cost estimates
5. Identify quantifiable and non-quantifiable benefits
6. Define alternative selection criteria
7. Compare alternatives
8. Report results and recommendations

Further information from the Army, at the center of their cost analyses, are:

• The analytic approach

• Cost-estimating methods
• Using current, constant, and discounted dollars
• Cost-estimating data sources
• Cost-estimation of software
• Cost as an independent variable
• Risk and uncertainty analysis
• Sensitivity analysis
• Validation analysis
• Cost-estimating errors
• Inherited assets
• Residual or salvage value

Finally, in this chapter, we have the categories of cost in a life cycle cost
estimate from the Defense Acquisition University (DAU):

a. Research and development

b. Investment
c. Operating and support
d. Disposal

Chapter 8 deals with everyday cost-effectiveness considerations. Evaluation

frameworks are examined for:

a. A simple computer system, and

b. A home “system”

The notion of incremental capabilities and cost is introduced for a typical CD

set of configurations and also several Honda automobiles. Increasing levels of
capability is an important topic that finds its way into everyday life.
Chapter 9 addresses selected military systems. Systems considered are:
 11 • Summary 99

a. The C-17
b. Navy Cargo Aircraft
c. The Joint Strike Fighter (F-35)
d. National Missile Defense, including subordinate elements of
• The Aegis Ballistic Missile Defense
• The Terminal High Altitude Area Defense
• The Space-Based Infrared System
• The Patriot Advanced Capability
• The Ground-Based Midcourse Defense

This is followed by a brief look abt effectiveness in space, and the MOEs for
the Cheyenne Mountain facility with respect to:

a. Response time
b. Information friend or foe
c. Probability of detection
d. Number of false alarms
e. State(s) of readiness

Various budget considerations follow, to include cost growth and a note on

transparency.
Chapter 10 deals with a variety of miscellaneous topics, starting with risk
analysis and the vaccine issue. This is comparing the risk of just a few cases
of blood clots against the benefits of the vaccine itself.
This chapter continues with a discussion of what is called a “business
case assessment,” which has the following typical elements:

1. Summary and recommendations

2. Key assumptions
3. Costs
4. Benefits
5. Cash flow projections
6. Risk
7. Strategic options
8. Opportunity costs

This chapter concludes with brief discussions of budgeting and connections to

the field of systems engineering.
Index

absolute values, 19–20 functional decomposition, 17–18, 67–68

analysis of alternatives (AoA), 26–27
Army War College, 87 incremental, capabilities and cost, 71–72
availability, 1, 93, 99 intercity transportation, 88
Aviation Advisory Commission, 16 International Council of Systems Engineering
(INCOSE), handbook, 28–29
budgeting, 85–86
business case assessment, 84–85 Joint Agency for CERs, 55–56
Joint Strike Fighter (JSF), 75–76
capability, 1, 93, 99
COCOMO, 52–54 life cycle costing, 43
cost analysis, guidance, 48 linear regression, 54–55
cost analysis, real meaning, 20–21
cost-benefit analysis, 37–38, 59 McNamara, Robert, 6–7, 32
cost categories, 44–45 measures, of effectiveness, 14–15, 59–60
cost-effectiveness, 31; approach to military cost-benefit, 61–64; budget, 90–92;
architecting, 25–26; bottom line, examples, 73–74
37–38; critique, 33; everyday, 67; miscellany, 83
history, 33; home, 69–70; principles MITRE engineering guide, 28, 57–58, 90
and controversies, 34; relationship to
systems engineering, 40; selected NASA, 89–90
commentaries, 31; selected National Missile Defense, 77–78
courses, 35–37 NIST handbook, 47–48
cost estimating relationships (CERs), 51–52
Office of the Secretary of Defense (OSD),
Defense Acquisition University (DAU), cost overall atmosphere, 34–35
analysis, 64–65 open-ended program design, 38–39
defense budget cutting, 79–81 O&S cost guide, 56–57
defining alternatives, 39–40
Department of Defense (DoD) budget, 48–49 planning, programming, and budgeting system
dependability, 1, 93, 99 (PPBS), 1
disposal costs, 66
RAND Corporation, 9–11
effectiveness, 1; measures, 15–16; in rational choice theory, 89
space, 78–79 risk analysis, 83–84, 89–90
ensemble, of prior systems, 13–14
Enthoven, Alain, 1, 6 sensitivities, 18
evaluation criteria, 15 summary, 93

101
102 Index

systems engineering, 23–24, 40 trends, 7

Systems Engineering Body of Knowledge
(SEBoK), 24–25 warfighting, 86–87
Whiz Kids, 1, 5–6, 8–9
transparency, 81 WSEIAC, 1–2, 4–5; recommendations, 4