Southeast Transportation Consortium

Final Report 636

Development of a Guidebook for

Determining the Value of Research Results
by

Yoojung Yoon, Ph.D.

Fei Dai, Ph.D.
Charles Wong, Graduate Research Assistant
Tse-Huai Liu, Graduate Research Assistant

West Virginia University

Published by:

4101 Gourrier Avenue | Baton Rouge, Louisiana 70808 | (225) 767-9131 | www.ltrc.lsu.edu
 TECHNICAL REPORT STANDARD PAGE

1. Title and Subtitle 5. Report No.

Development of a Guidebook for Determining the Value FHWA/LA.17/636
of Research Results 6. Report Date

2. Author(s) January 2021

Yoojung Yoon, Fei Dai, Tse-Huai Liu, Charles Wong 7. Performing Organization Code

3. Performing Organization Name and Address LTRC Project Number: 16-1PF

Department of Civil and Environmental Engineering SIO Number: DOTLT1000090
West Virginia University 8. Type of Report and Period Covered
Morgantown, WV 26506-6103 Final Report
4. Sponsoring Agency Name and Address January 2016-December 2018
Louisiana Department of Transportation and Development 9. No. of Pages
P.O. Box 94245 134
Baton Rouge, LA 70804-9245
Southeast Transportation Consortium

10. Supplementary Notes

Conducted in Cooperation with the U.S. Department of Transportation, Federal Highway
Administration
11. Distribution Statement
Unrestricted. This document is available through the National Technical Information Service,
Springfield, VA 21161.
12. Key Words
Research program; Mapping table; STC
13. Abstract
Measuring the value of research is vital for transportation agencies to justify the process efficiency and quality
of state research programs. Also, there is a growing need for aggregating the research benefits of each state at
the national level in order to develop and support requests for future federal funding for transportation
research. The objective of this project is to develop a guidebook that will provide a consistent approach for
measuring and documenting the value of completed research for the use of all STC research sections. To
obtain this goal, data were collected through a comprehensive literature review and nationwide survey to state
DOTs in the United States. Data analysis was conducted to identify research categories/subcategories, benefit
categories/subcategories, and benefit measures. A mapping table was then developed to allow users to
determine data types associated with the benefit measures. Finally, a quantification method was developed that
can calculate the benefit of each completed research. As a result, state DOTs will be able to use a scalable,
flexible, and consistent method to compare the value of various completed research projects. The significance
of these findings also serves as a baseline for state DOTs whom previously had no indications of the progress
of their projects towards their targets.
 Project Review Committee

Each research project will have an advisory committee appointed by the LTRC Director. The
Project Review Committee is responsible for assisting the LTRC Administrator or Manager in the
development of acceptable research problem statements, requests for proposals, review of
research proposals, oversight of approved research projects, and implementation of findings.

LTRC appreciates the dedication of the following Project Review Committee Members in
guiding this research study to fruition.

LTRC Administrator/Manager
Kirk Zeringue
Special Studies Research Manager

Members
Cindy Smith
Jarrod Stanley
Juanita Owens
David Sherman
Yu-Jen Chen
Terry Swygert
Binh Bui

Southeast Transportation Consortium

The RAC Region II has initiated a collaborative research program consortium through the Transportation
Pooled Fund (TPF) Program. The research program is called the Southeast Transportation Consortium
(STC) and is intended to encourage coordination among member states, as well as provide resources and
management of collaborative studies. The Consortium intends to address high priority transportation
research topics of common interest to the southeastern and adjoining states. Louisiana serves as the lead
agency in the STC.

Directorate Implementation Sponsor

Christopher P. Knotts, P.E.
DOTD Chief Engineer

— 2 —
Development of a Guidebook for Determining the Value
of Research Results

By
Yoojung Yoon, Ph.D.
Fei Dai, Ph.D.
Charles Wong, Graduate Research Assistant
Tse-Huai Liu, Graduate Research Assistant

West Virginia University

Morgantown, WV

LTRC Project No. 16-1PF

SIO No. DOTLT1000090

conducted for
Southeast Transportation Consortium
Louisiana Department of Transportation and Development
Louisiana Transportation Research Center

The contents of this report reflect the views of the author/principal investigator who is
responsible for the facts and the accuracy of the data presented herein.

The contents of do not necessarily reflect the views or policies of the Louisiana
Department of Transportation and Development, the Federal Highway Administration or
the Louisiana Transportation Research Center. This report does not constitute a standard,
specification, or regulation.

January 2021

— 3 —
 Abstract

Measuring the value of research is vital for transportation agencies to justify the process
efficiency and quality of state research programs. Also, there is a growing need for
aggregating the research benefits of each state at the national level in order to develop
and support requests for future federal funding for transportation research. However, only
a small number of state Departments of Transportation (DOTs) currently use
quantification procedures to determine the value of transportation research. According to
the results of a survey conducted by the latest Southeast Transportation Consortium
(STC) study, “STC Synthesis of Best Practices for Determining the Value of Research
Results,” a minimal number of state DOTs utilize evaluation guidelines, although almost
70% of the respondents to the survey answered that they have tried to evaluate the
quantitative and qualitative benefits of their research projects.

The major obstacles that prevent state DOTs from developing research value evaluation
approaches include: data scarcity for the measures selected to estimate research benefits;
difficulty in accurately describing the intangible benefits and interpreting the qualitative
benefits; unknown benefits at the time of a research value evaluation, which are
quantifiable only years after the research is completed; the diversity of the attributes of
research projects, which requires a variant form of the methods and measures for
estimating the research benefits; and different perspectives of grasping the value of
research exists between the interested groups (e.g., public agencies, political leaders,
communities, and researchers). Therefore, it is important to provide solutions to
overcome, or at least minimize, the described obstacles through the application of
alternative measures and quantification methods based on the availability of data and the
research output types to successfully develop a systematic, transparent, and practical
guidebook for state DOTs.

The objective of this project is to develop a guidebook that will provide a consistent
approach for measuring and documenting the value of completed research for the use of
all STC research sections. To obtain this goal, data were collected through a
comprehensive literature review and nationwide survey to state DOTs in the United
States. A thematic analysis was applied on the collected data to capture the meaningful
patterns of three main categories: Research Category, Benefit Category, and Benefit
Measures. These categories eventually developed a quantifiable value for all the data
types. A mapping table was then developed to allow users to determine a data type. These

— 4 —
values were then used to determine the benefit of the research by using them in a
quantification method. The quantification method has a range of cases that can occur
based on the amount of information available in the project. This information represents
the availability of historical data, target value, and performance value, which ultimately
will define the accuracy of the value of research.

Finally, a quantification method was developed that can calculate the benefit of each
completed research. As a result, state DOTs will be able to use a scalable, flexible, and
consistent method to compare the value of various completed research projects. The
significance of these findings also serves as a baseline for state DOTs whom previously
had no indications of the progress of their projects towards their targets.

— 5 —
 Acknowledgments

The funding for this guidebook was provided by the Southeast Transportation
Consortium (STC) and the Louisiana Transportation Research Center (LTRC) under SIO
No. DOTLT1000090 and LTRC Project No. 16-1PF. The authors of this guidebook
gratefully acknowledge the support and cooperation provided by the technical advisory
committee (TAC) of STC and the LTRC staff, especially Kirk Zeringue and Tyson
Rupnow, Ph.D. We also thank Daniel Vance and Eric Starliper of West Virginia
University for their contributions to this guidebook.

— 6 —
 Table of Contents

Technical Report Standard Page ..........................................................................................1

Project Review Committee ......................................................................................2
LTRC Administrator/Manager .................................................................................2
Members ..................................................................................................................2
Directorate Implementation Sponsor .......................................................................2
Development of a Guidebook for Determining the Value of Research Results...................3
Abstract ....................................................................................................................4
Acknowledgments....................................................................................................6
Table of Contents .....................................................................................................7
List of Tables ..........................................................................................................10
List of Figures ........................................................................................................ 11
Introduction ............................................................................................................13
Literature Review...................................................................................................15
Utah DOT—Measuring the Benefits of Transportation Research in Utah 17
Florida DOT - Review, Analyze and Develop Benefit Cost/Return on
Investment Equations, Guidelines and Variables .................................19
NCHRP-Web-Only Document 127: Performance Measurement Tool Box
and Reporting System for Research Programs and Projects ................21
Minnesota DOT - Benefits of MnROADPhase-II Research ......................22
New Jersey DOT—Research Implementation Report ...............................23
Utah DOT - Measuring the Benefits of Transportation Research in Utah .24
Ohio DOT—Evaluation of ODOT Research and Development
Implementation Effectiveness ..............................................................25
Texas DOT—2016 University Handbook-Chapter 6 ................................26
Alabama DOT—An Evaluation of the Benefits of the Alabama Service
and Assistance Patrol ...........................................................................27
California DOT—Mobile Work Zone Barriers ..........................................28
FHWA—Rural Road Low Cost Safety Improvements ..............................29
Florida DOT—Operational and Safety Impacts of Restriping Inside Lanes
of Urbane Multilane Curbed Roadways to 11 Feet or Less to Create
Wider Outside Curb Lanes for Bicyclists ............................................29
Florida DOT—Review, Analyze and Develop Benefit Cost/Return on
Investment Equations, Guidelines and Variables (Example 1) ............30

— 7 —
 Florida DOT—Review, Analyze and Develop Benefit Cost/Return on
Investment Equations, Guidelines and Variables (Example 2) ............31
Florida DOT—Review, Analyze and Develop Benefit Cost/Return on
Investment Equations, Guidelines and Variables (Example 3) ............32
Louisiana DOTD—Mechanistic Flexible Pavement Overlay Design
Program ................................................................................................33
Louisiana DOTD—Development and Performance Assessment of an FRP
Strengthened Balsa-Wood Bridge Deck for Accelerated
Construction .........................................................................................34
Louisiana DOTD—Investigation of Low Temperature Cracking in Asphalt
Pavements (Phase-II) ...........................................................................35
Illinois DOT- Development and Application of Safety Performance
Functions for Illinois ............................................................................35
Iowa DOT—Winter Operations Geographic Positioning Systems and
Automatic Vehicle Location ................................................................36
NCHRP—Performance Measurement Tool Box and Reporting System for
Research Programs and Projects (Example 1) .....................................37
NCHRP—Performance Measurement Tool Box and Reporting System for
Research Programs and Projects (Example 2) .....................................38
Objective ................................................................................................................39
Scope ......................................................................................................................40
Methodology ..........................................................................................................41
Data Collection (Literature and Discovery Search) ...................................41
Development of Research Categories and Subcategories ..........................43
Development of Benefit Categories/Subcategories and Measures ............44
Gap Analysis ..............................................................................................45
Evaluation Method .....................................................................................46
Discussion of Results .............................................................................................47
Data Collection through Surveys ...............................................................47
Research Categories and Subcategories.....................................................61
Benefit Categories/Subcategories and Measures .......................................69
Gap Analysis ..............................................................................................76
Utilization of the Mapping Table ...........................................................................78
Research Categories and Subcategories.....................................................79
Data Type(s) through the Intermediate Steps .............................................80
Example: Navigating the Mapping Table ..................................................81
Evaluation Method to Determine Research Value .................................................88

— 8 —
 Examples of the Possible Cases .................................................................91
Various Discrete Rating Systems ...............................................................96
Conclusions and Recommendations ....................................................................103
Acronyms, Abbreviations, and Symbols ..............................................................105
References ............................................................................................................107
Appendix ..............................................................................................................109

— 9 —
 List of Tables

Table 1. Research categories provided by responded state DOTs .................................... 48

Table 2. Summary of document and links submitted by the state DOTs .......................... 50
Table 3. The information provided by state DOTs for question 1 of Survey-2 ................ 54
Table 4. Benefit categories of the responding state DOTs ................................................ 56
Table 5. Benefit measures given by state DOTs ............................................................... 57
Table 6. Challenges/constraints to use the measures ........................................................ 60
Table 7. Benefit areas captured from state DOTs’ mission statements ............................. 69
Table 8. Feasible benefit categories and subcategories .................................................... 73
Table 9. Benefit measures corresponding to benefit categories/subcategories ................. 74
Table 10. A List of complete, incomplete, and under-developed research subcategories . 77
Table 11. Research areas by STC ...................................................................................... 79
Table 12. All possible combinations of target value, outcome, and historical data .......... 88
Table 13. Tentative confidence levels for research performance or pace ......................... 93
Table 14. Five-point ratings or grades for different scores ............................................... 96
Table 15. Descriptive scales based on percentages ........................................................... 96
Table 16. Research evaluated for PtTs in year 2018 ......................................................... 97
Table 17. Weights for the RCs .......................................................................................... 98
Table 18. Weighted PtT values for RCs ............................................................................ 99
Table 19. Weights for the RSCs ........................................................................................ 99
Table 20. Weighed mean PtTs at the level of RSC.......................................................... 100
Table 21. Weights for both RCs and RSCs ..................................................................... 101
Table 22. Weighted PtT values for the RC ...................................................................... 102

— 10 —
 List of Figures

Figure 1. Development of research categories and subcategories .................................... 43

Figure 2. Hypothetical example for steps to create benefit categories ............................. 45
Figure 3. Steps for gap analysis ........................................................................................ 46
Figure 4. State DOTs Participated in Survey-1 ................................................................. 47
Figure 5. State DOTs that responded to question 2 .......................................................... 50
Figure 6. State DOTs for present/future plans to develop quantification processes ......... 51
Figure 7. State DOTs (including DC) who agreed to follow-up in future ........................ 52
Figure 8. State DOTs (including D.C.) participated in survey-2 ...................................... 53
Figure 9. States that Responded to Survey 2 .................................................................... 54
Figure 10. State DOTs responding to question 2 of survey 2 ........................................... 55
Figure 11. State DOTs that provided benefit measures .................................................... 57
Figure 12. Map of state DOTs that provided challenges .................................................. 59
Figure 13. Research categories and subcategories based on type A ................................. 63
Figure 14. Research categories and subcategories based on type B ................................. 65
Figure 15. Research categories and subcategories based on type C ................................. 66
Figure 16. Example to estimate a coverage rate ............................................................... 67
Figure 17. Coverage rates of type A, B, and C for all state DOTs .................................... 68
Figure 18. Average coverage rates of three different category types ................................ 68
Figure 19. Proportions of keywords (Yoon and Dai 2017) ............................................... 71
Figure 20. Co-occurrence network of the keywords (Yoon and Dai 2017) ...................... 72
Figure 21. Steps to utilize the mapping table .................................................................... 78
Figure 22. Capture of the Mapping Table for the Design ................................................. 82
Figure 23. Determination of research categories .............................................................. 83
Figure 24. RCs and RSCs determined for the example research ...................................... 84
Figure 25. Measure and data type for erosion control ...................................................... 85
Figure 26. Data types under project efficiency measure ................................................... 86
Figure 27. Measure and data type for contractors’ performance on erosion control
practices .......................................................................................................... 86
Figure 28. Measure and data type for enhanced-lifespan erosion control devices ........... 87
Figure 29. Normal distribution curve of data points ......................................................... 90
Figure 30. Standard normal distribution showing Z-scores .............................................. 90
Figure 31. Historical data of average health index, research outcome, and target value .. 91
Figure 32. Research outcome and target value, but no historical data ............................. 92
Figure 33. Target value, but no research outcome and historical data .............................. 93

— 11 —
Figure 34. Research outcome and historical data, but no target value.............................. 94
Figure 35. Research outcome, but no historical data and target value.............................. 95
Figure 36. No research outcome, historical data, and target value .................................. 95

— 12 —
 Introduction

The mission of the U.S. Department of Transportation (DOT) and state DOTs can be
summarized as follows: providing a fast, safe, efficient, and environmentally-sound
transportation system in order to enhance the quality of life and facilitate economic
growth. To achieve this mission, the federal government allocated a total budget of
$90.98 billion, including the surface transportation reauthorization of $73.6 billion, to the
U.S. DOT in FY 2015 (U.S. DOT 2015). Also, according to the National Association of
State Budget Officers (NASBO), the average spending proportion of the states for
transportation in FY 2014 was estimated as 7.7% or approximately $34 billion (NASBO
2015). The research programs of state DOTs are intended to improve the understanding
of local, regional, and statewide problems in all areas of transportation and to find
solutions to those problems so that transportation agencies can establish more effective,
strategic planning based on the federal transportation appropriations. Therefore, the
mission of state transportation research programs generally includes, at a minimum,
promoting safety; reducing congestion and improving mobility; preserving the
environment, preserving the existing transportation system; improving the durability and
extending the life of transportation infrastructure; and improving goods movement (23
U.S. Code 508). For each purpose, state DOTs develop research topics, which may
employ fundamental, applied, and/or social science research as well as develop and apply
new technologies and federal and state agencies’ budgets. According to a report prepared
by the Office of Management and Budget (OMB), the federal budget proposed for
transportation research and development for FY 2017 is $1,065 million to advance the
nation’s strategic goals for a transportation system (OMB, 2016).

Every year, the U.S. Department of Transportation (DOT) and state DOTs fund millions
of dollars in research to cope with transportation issues, improve the quality of lives, and
facilitate economic growth. As a key to the research funding management, measuring the
benefits of the research results is indispensable to justify the efficiency and quality of
their research programs (Zmud, Paasche, Zmud, Lomax, Schofer, & Meyer, 2009; Ellis,
Degner, O’Brien, & Peasley, 2003; Concas, Reich, & Yelds, 2002; Hartman, 2001;
Anderson, 2010; Worel, Clyne, & Jensen, 2008; Sabol, 2001). Unfortunately, many state
DOTs have not truly measured the impacts of their transportation research projects on the
transportation system due, in large part, to a lack of comprehensive and implementable
quantitative and/or qualitative methods to determine the value of transportation research
projects (Schuler, 2010). A recent survey’s results revealed that 88% of the state DOTs

— 13 —
did not have any guideline or method to evaluate the quantitative or qualitative benefits
of their research projects; further clarification indicated that the 12% remaining that
provided responses misunderstood the questions and also did not have any guideline or
method (Ashuri, Shahandashti, & Tavakolan, 2014). As a result, a scalable, flexible, and
consistent method is missing that applies benefit measures and quantification procedures
considering data availability for measuring and documenting the value of the completed
research projects.

— 14 —
 Literature Review

A comprehensive literature review for existing practices, guidelines, and publications was
conducted to collect the information required for determining the value of transportations
research results. The main data collected include:

• Research categories and the criteria to classify future research projects

• Research output types which define the measures for research value
quantification
• The mission/objectives of a state research program
• Benefit categories and measures for each benefit
• Data types required to estimate measures
• Best and alternative evaluation methods based on the availability of data
• Methods to determine research value based on the evaluated benefits
• Documenting methods for easy communication

The data collected were synthesized for the five main data types such as research areas,
benefit categories, benefit measures, source of data, and methods used to quantify the
value of research. First, the research areas at which most of the investigated state DOTs
categorize to develop research are:

• Traffic
• Transport safety
• Environment
• Road design
• Pavement and materials
• Geotechnical
• Bridge/structure
• Maintenance and facility preservation
• Hydraulic and hydrology

Second, the most frequent benefit categories used in the quantification procedures are:

• Improved mobility
• Improved safety
• Improved environment
• Customer satisfaction

— 15 —
 • Improved infrastructure
• Expedited project delivery
• Improved technology
• Improved knowledge

Third, the benefit measures that are frequently observed for the quantification procedures
include:

• Reduction in travel time delay

• Reduction in crashes
• Dollar savings due to reduction in crashes
• Reduction in vehicle emission
• Dollar savings due to reduction in emissions
• Dollar savings by offering free services to the customers
• Crash modification factors
• Benefit-cost ratio
• Reduced installation time of the bridge deck
• Dollar savings due to use of durable materials/products
• Dollar savings due to reduction in number of traffic signals
• Dollar savings due to use of less materials and resources
• Sponsored university students

Fourth, the state DOTs utilize the data required for the benefit measures from:

• Surveys to principal investigators and users

• Regular data inspection such as crash rates, traffic congestion, etc.
• Field experiments
• Engineering judgment based on historical data such as material unit cost,
value of time, etc.
• Data obtained from the simulation software

Lastly, the methods used by state DOTs to evaluate the research values are:

• Scaling techniques based on 3- or 5-point scales

• Benefit-cost estimation
• Dollar benefit analysis (savings due to materials or enhanced performance and
cost of avoidance)
• Benefit analysis based on experimental and before/after study

— 16 —
 • Computer simulations

Most of these data were found at the reports and guidelines of state DOTs. The
objectives, research categories, benefit categories and measures, and quantification
methods and required data of these public documents are summarized as follows:

Utah DOT—Measuring the Benefits of Transportation Research in Utah

Objectives

Utah DOT conducted a study to measure the benefits of transportation research in order
to justify the expenditure of transportation funding and to determine the most appropriate
use of the available budgets. The study objectives were as follows:

• Estimate the benefits of major research projects and compare them with the
costs to conduct the studies.
• Determine which types of projects produce the highest benefit-cost ratios and
which projects are more often unsuccessful or marginal.
• Provide information on the management and support of research projects.
• Make recommendations concerning the research program and the types of
projects undertaken in the future.

Benefit Categories and Benefit Measures

In general, benefits and corresponding measures are identified as follows:

• Asset Management: contribution ($) to the management of UDOT’s assets

• User Impacts: Reduction ($) in impacts to the travelling public.
• Safety: Improvement ($) in the safety of the travelling public, UDOT and/or
contractor employees.
• Quality of Life: Improvement ($) in the quality of life of residents and visitors
to the state, including aesthetic beauty, convenience, comfort, and security.
• Environmental: Improvement ($) in the quality of the natural environment.
• Level of Knowledge: Level of knowledge expansion ($) in proposed research
area.
• Administration and Policy: Improvement ($) in administrative, management,
and policy decisions.

— 17 —
Method and Data

• Form a Technical Advisory Committee (TAC): A TAC, comprised of research

managers and others who are likely to use the findings, is formed to provide
input during the program evaluation effort.
• Select Projects for Evaluation: Projects are selected for evaluation. An attempt
is made to evaluate all projects, but because some research is not fully
implemented immediately after the project is complete, it may be necessary to
allow a period of time between project completion and the assessment of the
benefits. If a project is noted as “benefits not known at this time”, the project
is re-visited during the next evaluation effort.
• Compile a List of Projects to be Evaluated: A list of projects to be evaluated is
compiled, including project title; key champion; project manager; project cost;
and deliverables received. Projects are classified into the following types:
Infrastructure Related Research, Operations Related Research, or Policy
Related Research (per Report No. UT-10.01 prepared in 2010, 41 projects
were evaluated, which were completed during 2006, 2007, and 2008).
• Evaluator Meets with Project Champions to Collect Benefits Data: For each
project, an evaluator meets with the key champion and others familiar with the
research products. A plan is outlined for estimating benefits and total costs. A
“Benefits Assessment Form” is used to collect and document benefits. The
evaluator guides the key champion through the evaluation process by
collecting input using the “Benefits Assessment Form” and calculating
benefits.
• Calculate Project Benefits: The evaluator calculates project benefits, using
data and input from the project champion. Assumptions and calculations are
conservatively estimated, in order to maintain a credible benefits value.
• Assign a Grade to Each Project: To assess the benefits as improved operations,
a grade is assigned to each project based on the following descriptions:
 A = Major impact- Enhanced operations (specification, policy,
standard, method, etc.)
 B = Significant impact- Improved operations
 C = Contributed to state-of-the-practice
 D = Unclear or contradicting findings- More study needed
 E = Major tasks not completed- Objectives not met
• Calculate Benefit-Cost Ratios: A benefit-cost ratio is calculated to assess cost
saving benefits for each individual project. In addition, benefit-cost ratios are

— 18 —
 calculated for each project type and for the total three-year period. Benefit-
Cost Ratio = Total Financial Benefit ($) / Total Project Cost ($).

Florida DOT - Review, Analyze and Develop Benefit Cost/Return on

Investment Equations, Guidelines and Variables

Objectives

According to this research report, the mission of the research program at Florida DOT
(FDOT) has been to improve and protect Florida’s transportation system through the
ethical scientific conduct of research that increases global knowledge of products,
processes, and practices to transfer information and to encourage the implementation of
research results. The objective of this research was to develop a flexible evaluation
system to assess the benefit of various research projects.

Research Categories

• Construction
• Environmental Management
• Materials and Testing
• Operations
• Planning
• Public Transit
• Roadway Design
• Safety
• Structural Engineering
• Traffic Operations

Benefit Categories

Qualitative benefits categories:

• Improvements to Knowledge Base

• Improvements to FDOT Infrastructure (Organizational and Process Structures)
• Improvements to Quality of Life
• Improvements to FDOT Management and Policy

— 19 —
Economic benefit categories that are utilized are as follows:

• Improved Work Efficiency

• Reduced Material Costs
• Reduced User Cost
• Reduced Maintenance Cost
• Reduced Construction Cost
• Reduced Operational Cost

Method and Data

• Develop a scoring system for qualitative benefits: The following numerical

scale is included in the initial evaluation form: “1” = Project Did Not Meet
Expectations, “2” = Project Meets Expectations, and “3” = Project Exceeds
Expectations. Evaluation consisted of scoring the project with regard to
specific benefit listing in each of the functional areas.
• Create benefit evaluation form: An initial research/benefit evaluation form
was created based on the investigation of current practice and literature. The
forms are distributed to the project Principal Investigators (PI) and the FDOT
project coordinators to jointly rate the project and indicate the project
contributions ($).
• Calculate benefit-cost: Benefit-cost is calculated based on feedback received
form the PI and project coordinator to determine the savings per unit by
comparing the cost prior to implementation with the expected cost after
implementation. Total savings is estimated by multiplying the unit savings by
the estimated total number of units. Future cost savings should be converted
to present values using appropriate interest rate values (this rate is the value
established and used by the FDOT for all planning calculations). Estimated
benefit-cost ratio = present value of total cost saving/present value of cost of
research.

— 20 —
 NCHRP-Web-Only Document 127: Performance Measurement Tool
Box and Reporting System for Research Programs and Projects

Objectives

The objective of this project was to create a research performance (RPM) system that
integrates a balanced and broadly applicable set of research performance measures and
tools to assist practitioners in applying these measures to their research projects and
programs. RPM system is composed of a web site, RPM-Web, and a complementing CD-
ROM tool box called RPM-Tools.

Benefit Measures

There were 20 research performance measures collected through a review of literature

relevant to transportation research and other associated research areas. These
performance measures included outcome, output, efficiency, resource allocation, and
stakeholder metrics. Performance measures are listed below:

• Return on investment or benefit-cost ratio

• Lives saved
• Construction, maintenance, and operations cost savings
• Reduction in crashes
• Reduction in system delays
• Positive environmental impact
• Quality of life enhancement
• Safety enhancement
• Level of knowledge increased
• Management tool or policy improvement
• Public image enhancement
• Technical practices or standards upgrades
• Leadership
• Percent of projects/products implemented
• Percent of projects completed on time
• Percent of projects completed within budget
• Number of contractors
• Number of contractor partnerships
• Percent of satisfied customers

— 21 —
 • Contribution to the overall mission of the department

Method and Data

RPM-Web offers the user three methodologies for estimating benefits. Every benefit
estimation method involves statistical data and the assumptions. The Resource Collection
described above is one source for necessary statistical information obtained from the
most knowledgeable individuals within the agency. The three methodologies RPM-Web
offers are:

• Current Minus Future Method: This method requires two determinations of

costs, fatalities, and/or numbers of crashes.
• Direct Difference Method: This method is particularly well-suited for use
when the research project provides estimated benefits per application of the
research product, or when the expected benefits per application can be
estimated after the research project is completed.
• Percent Improvement Method: This method is ideal when the research project
determines a percentage improvement to be expected in costs, fatalities and/or
numbers of crashes, or when a percentage improvement can be estimated after
the research project is completed.

Minnesota DOT - Benefits of MnROADPhase-II Research

Objectives

This project was to assess the benefits in the core areas including: design guide,
innovative construction, preventative maintenance, recycled materials, rehabilitation,
surface characteristics, and continued support of non-pavement research using the
MnROAD site.

Benefit Categories and Measures

• Economic lifecycle cost analysis based on construction costs and discount

rates;
• Cost benefit of avoidance (learning from mistakes)
• Savings in maintenance costs
• Reduced noise from quieter pavements (environmental cost)

— 22 —
 • Savings in materials costs
• Savings from extending pavement service life

Methods and Data

The methodology aims at analysis of the benefits based on various approaches listed
below:

• Direct: savings due to either materials or enhanced performance. Monetary

benefits are quantifiable estimates
• Indirect: new or streamlined construction processes that save time and/or
improvement to quality, improvements to performance due to enhanced
quality, benefits difficult to quantify
• Avoidance: learn from mistakes; MnROAD is more conducive to risk-taking;
benefits difficult to quantify; benefits obtained by avoiding the same mistakes
elsewhere; difficult to quantify
• Demonstration: technology transfer through demonstration of procedures,
process, or equipment; instill confidence in users to try something new;
difficult to quantify benefits

New Jersey DOT—Research Implementation Report

Objectives

The intent of this report was to identify the quantitative and qualitative benefits of New
Jersey DOT (NJDOT) research projects that were completed in 2007. Benefits were
defined as technology transfer; enhancements; cost savings and economic impact;
improvement of safety; and reduction of labor time for the customers, known as
champions.

Benefit Categories and Measures

• Enhancements (aesthetic, efficiency/effectiveness, environmental, and safety)

• Cost savings and economic impact
• Technology transfer

— 23 —
Method and Data

University principal investigators were interviewed by email, telephone, or in-person on

the outcomes of their research projects. It was expected that the investigators would
report on additional studies and projects that had been generated as a consequence of this
initial work. Since the information was not always available, NJDOT customers were
identified by the investigators for each project. Interviews were arranged with the
customers to establish not only benefits, but follow up activities that happened as a result
of the research. When required, NJDOT project managers were also contacted to provide
background information.

Utah DOT - Measuring the Benefits of Transportation Research in Utah

Benefit Measures

• Dollar savings
• Benefit-cost ratio

Method and Data

• Net present value that compares the amount invested today to the present
value of the future savings from the implemented research.
• Benefit-cost analysis for estimating the strengths and weaknesses of
alternatives that satisfy transactions, activities, or functional requirements for
a project. It is used to determine options that provide the best approach for the
adoption and practice in terms of benefits in labor, time and cost savings.
 Benefit-cost ratio attempts to summarize the overall value for money
of a project or proposal. BCR = Present Value of Total Savings
(Benefits) / Present Value of Cost of Research.
 Total Savings (Estimated Savings over a period of Years – Cost of
Implementation)
 Payback Period is calculated by counting the number of years it will
take to recover the budget invested in a project.
• Payback period (Years) = Project Cost / Cost Savings per Year.

— 24 —
 Ohio DOT—Evaluation of ODOT Research and Development
Implementation Effectiveness

Objectives

One major objective of this research project was to determine the extent of
implementation and effectiveness of research results; determine the dollar value of
benefits (if possible) and compare with costs.

Benefit Categories

• Dollar savings
• Benefit-cost ratio
• Project grade/score (qualitative measure)

Method and Data

Evaluation of the research project was done based on the following criteria:

• Implement ability: Assigning a grade between 0 and 100 to a project by the

researcher, the liaison representatives, or by the investigator after studying the
final report and the implementation forms. A 0 means the research projects
didn’t have any implementable findings, while the projects with fully
implementable results were assigned 100.
• Implementation: “A number between 0 and 100. This parameter was assigned
to each project in the same way as implement ability. ODOT staff interviews
were the main source of information for determining the findings that were
implemented.”
• Dollar Savings: “The investigator could determine the dollar savings for only
four research projects. The other projects in the sample did not have an
identifiable direct dollar benefit. The benefits of these projects were of a
qualitative nature.”
• Benefit-Cost: “Because of the lack of information concerning the dollar
benefits of the projects, a formal benefit-cost study applying engineering
economy techniques was not possible. However, the benefit-cost variable
according to the value judgment of the people involved in the research project

— 25 —
 was determined. This variable takes a value for any given project, from very
low to very high.”
• Project Success: “A successful project is one that has met its objectives, has
enjoyed wide user acceptance, or provided some technological information for
immediate use in the field or for later use in future research projects. A
number between 1 and 5 was assigned to each project, with 5 representing a
very successful project. The numbers were investigator's own interpretation of
ODOT staff comments recorded during the interviews and stated in the
documentation.”
• Implementation Effectiveness (IE): “Implementation effectiveness was
defined as the result of the implementation divided by the implement ability in
a percentage term. The mean value of the variable IE was 80. A 90%
confidence interval was computed and resulted in: 66 < mean (IE) < 94.”

Texas DOT—2016 University Handbook-Chapter 6

Benefit Categories and Measures

• Level of knowledge
• Management and policy
• Quality of life
• Customer satisfaction
• Environmental sustainability
• System reliability
• Increased service life
• Improved productivity and work efficiency
• Expedited project delivery
• Reduced administrative costs
• Traffic and congestion reduction
• Reduced user cost
• Reduced construction, operations, and maintenance cost
• Materials and pavements
• Infrastructure condition
• Freight movement and economic vitality
• Intelligent transportation systems
• Engineering design improvement

— 26 —
 • Safety

Alabama DOT—An Evaluation of the Benefits of the Alabama Service

and Assistance Patrol

Objective

The key objective of this study was to evaluate and demonstrate the economic benefits of
the Alabama Service and Assistance Patrol program (ASAP) to travelers in the
Birmingham region. These benefits include those associated directly with the motorist
assistance rendered as a part of ASAP, as well as indirect benefits associated with reduced
delay, improved safety, and reduced environmental impacts.

Research Category

• Traffic, transport safety, environment

Benefit Categories

• Improved mobility, improved safety, improved environment, customer

satisfaction

Data Type

• Travel time value, average vehicle occupancy

• Crash reduction rates from literature, economic value of crash reduction rate
from literature
• Vehicle miles traveled, vehicle type
• Vehicle miles traveled, vehicle type, reduction in emission obtained from the
simulation software, economic value of emission reduction
• The average value of service per assist, the number of assists per year

Method

• Traffic simulation
• Benefit analysis by converting accident reduction rates to dollar values
• Traffic simulation

— 27 —
 • Benefit analysis by converting accident reduction rates as well as emission
reduction to dollar value
• Benefit analysis by converting the number of assists per year to dollar savings

Measures

• Reduction in delay
• Dollar savings due to reduction in crashes
• Reduction in emission
• Dollar savings due to reduction in emission and accident cost
• Dollar savings by offering free services to customers

California DOT—Mobile Work Zone Barriers

Objective

To reduce the traffic delay and reduce traffic interruption

Research Category

• Traffic

Benefit Categories

• Improved Mobility

Data Type

• Field data (Mobile barrier set up and break down time)

Method

• Experimental

Measures

• Reduction in travel delay

— 28 —
 FHWA—Rural Road Low Cost Safety Improvements

Objective

To evaluate the safety effectiveness of several low-cost safety strategies presented in the
NCHRP

Research Category

• Road design

Benefit Categories

• Improved safety

Data Type

• Pavement width, lane width

Method

• Statistical analysis

Measures

• Crash modification factor (CMF)

Florida DOT—Operational and Safety Impacts of Restriping Inside

Lanes of Urbane Multilane Curbed Roadways to 11 Feet or Less to
Create Wider Outside Curb Lanes for Bicyclists

Objective

To evaluate safety and operational benefits of using wider outside lane than inside lane on
multiline roadways.

— 29 —
Research Category

• Road design

Benefit Categories

• Improved safety

Data Type

• Field data (crash data), inside lane width

Method

• Before after study, regression model

Measures

• Reduction in number of crashes

Florida DOT—Review, Analyze and Develop Benefit Cost/Return on

Investment Equations, Guidelines and Variables (Example 1)

Objective

Investigation has resulted in revision to current Superpave asphalt pavement material

specifications with regard to course aggregates. A revised and more flexible aggregate
specification will permit the use of some Florida aggregates and reduce the need for
imported materials. Local materials are available at less cost than the imported materials.
The result is a predicted savings in the cost of Superpave asphalt paving.

Research Category

• Materials

Benefit Categories

• Improved Infrastructure

— 30 —
Data Type

• Lower course aggregate price, Savings per TN of Superpave Asphalt, Total

Superpave Quantity in TNs, Present Value of the Future Savings I = 5%

Method

• Benefit cost analysis

Measures

• Benefit cost ratio

Florida DOT—Review, Analyze and Develop Benefit Cost/Return on

Investment Equations, Guidelines and Variables (Example 2)

Objective

This research has developed a wireless system for transmitting driving data of interest
from precast concrete piles while they are being driven. Sensors and a transmitting unit
are cast into the pile at the plant, thereby eliminating instrumentation and wiring the pile
at the project site. Transmitted data is received on a portable computer at the site. Driving
and capacity analysis can be made in real time. Currently, the construction contractor is
paid to drive the test piles at a higher cost than regular production piles because of the
additional time required. The product of this research is a test pile that can be installed at
the same driving cost as a regular production pile.

Research Category

• Geotechnical

Benefit Categories

• Improved Infrastructure

— 31 —
Data Type

• The production unit of Lineal Feet (assuming a 40-foot pile), Production Pile
Average Unit Price, Quantity of Test Piles, Present Value of Future Savings, I
= 5%

Method

• Benefit cost analysis

Measures

• Benefit cost ratio

Florida DOT—Review, Analyze and Develop Benefit Cost/Return on

Investment Equations, Guidelines and Variables (Example 3)

Objective

The results of this study suggest that current noise barrier wall criteria may be modified
to provide for a lower wall height that will also meet noise control requirements.
Research recommendations indicate that barrier wall heights can be reduced by 1.5 feet.
Lower wall heights are expected to result in lower construction cost for new walls.

Research Category

• Environment

Benefit Categories

• Improved environment

Data Type

• The production unit of square feet of noise barrier wall, the average unit cost
for noise barrier wall, present value of future savings, I = 5%

— 32 —
Method

• Benefit-cost analysis

Measures

• Benefit-cost ratio

Louisiana DOTD—Mechanistic Flexible Pavement Overlay Design

Program

Objective

To develop an overlay design method/procedure that is used for a structural overlay

thickness design of flexible pavement in Louisiana based on In-situ pavement conditions
and Non-destructive test (NDT) methods.

Research Category

• Pavement and material

Benefit Categories

• Improved infrastructure

Data Type

• Pavement mile, material quantity, pavement life, construction cost

Method

• Benefit cost analysis

Measures

• Benefit cost ratio

— 33 —
 Louisiana DOTD—Development and Performance Assessment of an
FRP Strengthened Balsa-Wood Bridge Deck for Accelerated
Construction

Objective

To develop, construct, and evaluate a lightweight fiber-reinforced plastic (FRP)-wrapped

balsa wood bridge deck system

Research Category

• Bridge/structure

Benefit Categories

• Expedited project delivery

• Improved infrastructure

Data Type

• Field data

Method

• Simulation by finite element analysis

Measures

• Reduced installation time (number of days)

• Dollar savings due to durable materials

— 34 —
 Louisiana DOTD—Investigation of Low Temperature Cracking in
Asphalt Pavements (Phase-II)

Objective

To develop the test methods and specification criteria that will allow the selection of
fracture resistant asphalt mixtures and binders at low temperatures

Research Category

• Pavement and material

Benefit Categories

• Improved infrastructure

Data Type

• Pavement lane mile, average cost of pavement per mile number of cracks

Method

• Benefit dollar analysis by comparing the new and traditional pavements

Measures

• Dollar savings due to use of durable pavement

Illinois DOT- Development and Application of Safety Performance

Functions for Illinois

Objective

To evaluate and implement state specific safety performance functions (SPFs). SPFs
provide a realistic and accurate prediction of crash frequency, severity, type, etc. This
allows IDOT to identify high incident areas and decide which areas are the best
candidates for safety improvements

— 35 —
Research Category

• Environment

Benefit Categories

• Improved safety

Data Type

• Crash data, average annual daily traffic (AADT), beginning station, county,
ending station, functional class, inventory (key route), median type, number of
lanes, segment length, township, urban code

Method

• Statistical analysis

Measures

• Reduction in crashes

Iowa DOT—Winter Operations Geographic Positioning Systems and

Automatic Vehicle Location

Objective

To evaluate the performance of the trucks equipped with Geographic Positioning Systems
and Automatic Vehicle Location during winter

Research Category

• Environment

Benefit Categories

• Improved safety/improved mobility

— 36 —
Data Type

• Discount rate, life cycle, number of vehicles installed with AVL, total number
of vehicles, loaded labor cost per hour, lane miles covered per storm (per
truck), annual number of storm events, average labor hours per storm event
(per vehicle), operating cost per mile (excluding labor), estimated minutes
doing paperwork per storm (per vehicle), total storm event crashes (per
season), average cost per crash

Method

• Benefit-cost analysis

Measures

• Benefit-cost ratio

NCHRP—Performance Measurement Tool Box and Reporting System

for Research Programs and Projects (Example 1)

Objective

• Traffic signal warrant verification

Research Category

• Traffic engineering

Benefit Categories

• Improved mobility

Data Type

• Discount rate, average cost of installation of traffic signal, estimated number

of reduction in traffic signals

— 37 —
Method

• Benefit analysis by comparing before/after situation

Measures

• Dollar savings due to reduction in the number of traffic signals

NCHRP—Performance Measurement Tool Box and Reporting System

for Research Programs and Projects (Example 2)

Objective

To evaluate structural steel design tool

Research Category

• Structure

Benefit Categories

• Improved infrastructure

Data Type

• Anticipated life of product, discount rate, average composite salary rate of

engineering staff ($50/hr.), average number of steel structure designed,
number of hours of engineering design

Method

• Benefit analysis

Measures

• Dollar savings due to number of hours saved per steel structural design

— 38 —
 Objective

The primary objective of this research was as follows:

• To develop a guidebook that will provide a consistent approach for measuring

and documenting the value of completed research for the use of all STC
research sections.

To achieve the research objective, the specific aims of the work proposed were therefore
as follows:

• Investigate all possible aspects (e.g., state DOT organizational structures,

state/national transportation missions, research objectives, and research
attributes such as qualitative or quantitative) to develop a list of research
project categories in a hierarchical structure and to prepare the criteria for
determining the research types of future projects.
• Define the parameters required for determining the values of research projects
in relationship tables/diagrams.
• Develop a straightforward decision matrix to guide public agencies from a
starting point (research categories) to an end point (measure quantification
methods) with examples.
• Develop a rating method to determine research values by integrating all of the
qualitative and quantitative measures.

— 39 —
 Scope

The synthesis concluded that the need exists to develop a systematic and transparent
approach to determining the value of transportation research. The approach should be
scalable, flexible, and easy to understand. A guidebook will be required to accommodate
and measure the value of research from multiple types of projects. The guidebook needs
to incorporate a well-defined process that delivers results in a simple and easily applied
manner.

Through this project, a consistent method for determining the value of research will be
developed with the understanding that research categories and subcategories are very
diverse. The focus will be to develop various measures in order to easily evaluate the
majority of the research performed by state DOTs all around the nation. The process for
determining the measures will be done to ensure maximum coverage of all research areas
done throughout the STC. Because of the uniqueness of this project, multiple steps will
be taken that allow the STC to evaluate selected phases before moving on to the next
phase. All the preliminary steps lead into the development of a guidebook that provides
guidelines and worked examples for the use of the STC members.

— 40 —
 Methodology

Data Collection (Literature and Discovery Search)

The aim of the literature review and discovery search was to review the existing
practices, guidelines, and publications in order to collect information required for
determining the value of transportations research results. The primary data collected
through the research activities include:

• Research categories and the criteria to classify future research projects.

• Research output types which define the measures for research value
quantification.
• The mission/objectives of a state research program.
• Benefit categories and measures for each benefit.
• Data types required to estimate measures.
• Best and alternative evaluation methods based on the availability of data.
• Methods to determine research value based on the evaluated benefits.

To collect the information described above, the literature review and discovery search
first focused on the publications of completed and ongoing studies conducted by state
DOTs, federal agencies, and private research institutions as well as journals that relate to
determining the value of research. The discovery search also visited websites of all U.S.
research programs (e.g., the second Strategic Highway Research Program [SHRP2],
National Cooperative Highway Research Program [NCHRP], and Every Day Counts
[EDC] initiative) and state DOTs to understand research areas and retrieve mission
statements, which were utilized to develop the benefit categories. This task also required
conducting surveys on state DOTs to investigate the existing practices and procedures as
well as future plans to evaluate research values. Two separate consecutive nationwide
surveys followed to obtain corresponding and missing data. The information collected by
the survey is important for the gap analysis in order to develop a comprehensive,
practical, and compatible guideline for state DOTs to determine research values.
Therefore, the survey questions were developed based on, at a minimum, the following
information:

• Documenting methods for easy communication.

• A list of the research categories in use in a state DOT.

— 41 —
 • Well- and ill-established categories in terms of the quantification processes
suggested in this proposal.
• Reasons for the well- and ill-established categories, for example:
o Data availability (e.g., data existence, difficulties in interpreting)
• Resource (e.g., staff, technique, system) availability
• Research categories that should be further developed

Survey-1 was designed to capture the states’ knowledge and practices in determining the
value of transportation research, which included information such as the research
categories used by state DOTs, the quantification process for measuring completed
research values, and the current and future plans of state DOTs to develop quantification
processes. To collect the information, a total of five questions were developed. The
questions were carefully reviewed by the technical advisory committee (TAC) of this
research. A web-based questionnaire platform was utilized for the questions, which
helped increase the response rates. The list of surveyees for Survey-1 was obtained from
the American Association of State Highway and Transportation Officials Research
Advisory Committee (AASHTO RAC) website to whom the link to the web-based survey
questionnaire was. The questions developed are as follows:

• Does your organization categorize research projects in research areas?

• Have you ever developed any quantification process to measure and document
the value of completed research?
• Would you please provide the link or attach the related documents?
• Do you have any present/future plans to develop any quantification process to
determine the value of research benefits?
• May we follow up with you to obtain more information in the future?

Survey-2 was a follow-up survey to collect additional information, such as criteria for
development of research categories, benefit categories, benefit measures, and similar
challenges for use in gap analysis and development of quantification procedures. Similar
to Survey-1, all the questions in Survey-2 were carefully reviewed by the TAC of this
research, and a web-based questionnaire platform was utilized for the questions. The
surveyees list used for Survey-1 was also used to deliver Survey-2. To collect the
information for Survey-2, a total of four questions were developed as listed below:

• If your organization has research categories to develop research projects and

uses criteria to classify the research categories, please list the criteria in the
textbox below; otherwise, skip this question.

— 42 —
 • If your organization has ever used benefit categories (e.g., improved safety,
improved mobility, etc.) to determine the value of transportation research
results, please list them in the box below; otherwise, skip this question.
• If your organization has ever used qualitative/quantitative measures for each
of the benefit categories listed in question 2, please list them in the box below;
otherwise, skip this question. (You can write up to three measures for each
benefit category)
• If your organization has ever experienced any challenge/constraints to using
the measures, please explain the challenges/constraints in the box below.

Both surveys were implemented to fulfill the goal of the entire project. Only the data
about the present study from the surveys were used.

Development of Research Categories and Subcategories

The purpose of developing research categories and subcategories was to standardize

research areas being used by state DOTs. In the traditional categorization approach,
categories should be clearly defined based on the similar properties of objects to be
grouped, mutually exclusive, and collectively exhaustive (Johnson and Christensen,
2014). To address the three characteristics of developing categories, a hierarchical
structure for the research categories needs to be constructed. The categories in a
hierarchical structure are then organized into different levels in which discrete
subcategories share a set of properties of a higher-level category. Therefore, as shown in
Figure 1, the procedures to develop research categories and subcategories were utilized
for maximum coverage of the most researched areas in a hierarchical classification.

Figure 1. Development of research categories and subcategories

— 43 —
First, based on the information collected in the literature and discovery search, the
research areas that were in a nonhierarchical structure were identified, and the
descriptions for the research areas were developed to capture the meaningful patterns in
each research area through thematic analysis. The captured patterns were utilized to
define the horizontal and vertical relationships among the research areas to develop an
initial hierarchical category structure by clustering and coding. Coding is the process of
assigning symbols or names to clustered research areas for categorization (Miles and
Huberman, 1994). Then, the credibility of the initial category structure was tested with a
sample of actual research projects in order to answer the following critical questions:

• How distinct is each category from the other categories?

• How inclusive is the hierarchical category structure for the maximum
coverage of the research areas developed among the state DOTs in the STC?
• How simple is the hierarchical category structure?

If the initial hierarchical category structure was found to meet the criteria, the research
activities in this task proceeded to the final hierarchical category structure; otherwise, the
second and third research activities in Figure 1 were repeated until the questions were
adequately addressed.

Development of Benefit Categories/Subcategories and Measures

In general, a research value is an evaluation of the impact (called outcomes) of research

viewed within a narrowly defined community (Hofmeister, 2011). Measures can be
defined as the metric used to determine research values. The success of research
programs is supported by high-value research projects. Therefore, the development of a
list of measures started by identifying all possible objectives, which were redefined and
grouped to create new “benefit categories and subcategories” that were suggested to
clearly understand the properties of the measures. Benefit categories and subcategories
are the keys to the development of benefit measures. We utilized thematic analysis, as
well as clustering and coding approaches, to generate benefit categories and
subcategories. Figure 2 graphically presents the steps to create benefit categories. Florida
DOT’s research program aims to increase the safety, reliability, capacity, and efficiency
of the state transportation system. The research program of Oregon DOT intends to
enhance the performance (e.g., safety, infrastructure management practices, innovative
contracting and project delivery, and sustainable environmental practices) of the state’s

— 44 —
transportation system. The thematic analysis captures a set of words from the two state
DOTs, and then these words are clustered and assigned by category names.

Figure 2. Hypothetical example for steps to create benefit categories

Benefit measures are generally defined as functional key performance indicators to

monitor the operation and management progress of transportation systems. Consequently,
the benefit measures are also utilized as the indicators to make well-informed decisions to
establish a long-term strategic goal for safe and reliable transportation systems. State
DOTs develop various research initiatives to overcome the current and future challenges
in transportation systems by introducing cost-effective technologies/techniques and
innovative, cutting-edge knowledge. It implies that the benefit measures to evaluate the
performance of transportation systems also should be applied to determining the value of
transportation research. Therefore, various benefit measures were investigated through a
comprehensive literature review and the analysis of the data obtained from Survey-2,
considering the benefit categories and subcategories.

Gap Analysis

The gap analysis was conducted on the existing research value assessment processes
being used by state DOTs. Three separate lists of research subcategories based on the
existence of formal guidelines and examples to determine the value of research projects
are as follows: 1) research subcategories with formal procedures (well-established
research subcategories), 2) research subcategories with procedures but need to be
reproduced (incomplete research subcategories), and 3) research subcategories without
formal procedures (under-developed research subcategories). The gap analysis was

— 45 —
conducted based on the results in the previous steps (i.e., development of research
categories/subcategories and benefit categories/subcategories) by comparing the project
categories and corresponding measures with existing quantification processes and
examples already developed by other state DOTs or agencies. Therefore, the method for
gap analysis consisted of the following three steps as shown in Figure 3.

Figure 3. Steps for gap analysis

Evaluation Method

The development of an evaluation method is essential to assist state DOTs in assessing

the value of completed research based on the data types estimated. Computing the value
of research is vital, and DOTs have struggled to justify the process efficiency and quality
of state research programs. There is an increasing need for a compatible measuring scale
of research benefits in order to develop and support requests for future federal funding for
transportation research (Krugler et al. 2006). Therefore, an evaluation method was
developed considering the aspects as follows:

• The evaluation method should be easy in the sense of the development of the
equation that would finally be used to calculate the research values of
projects.
• The evaluation method should be simple enough to be applied throughout a
wide range of DOT projects.
• The evaluation method should be capable of encompassing many variations of
state DOT projects to create a universal compatible measuring scale.

— 46 —
 Discussion of Results

Data Collection through Surveys

Survey-1 was conducted to capture the state of knowledge and practice in determining
the value of research in DOTs, while Survey-2 was conducted to collect the examples of
benefit categories and corresponding measures used by the state DOTs. The screenshots
of both surveys and corresponding responses are included in Appendix A1 and A2 for
Survey-1, and Appendix B1 and B2 for Survey-2.

Survey-1 Statistics

A total of 27 state DOTs finally participated in this survey as graphically displayed in

Figure 4. The District of Columbia DOT participated in this survey but is not displayed in
the map due to its size.

Figure 4. State DOTs Participated in Survey-1

Question 1: Does your organization categorize research projects in research areas?

Out of 27 respondents to the first survey, eighteen individuals provided the list of
research categories that they use in their organization (Table 1). The response rate to the
survey of all 50 state DOTs is 54%. Table 1 represents the research categories provided
by the responded state DOTs. It is very impressive that Washington DOT utilizes very

— 47 —
granular research categories compared to other state DOTs. It is also interesting to
observe that the Louisiana DOTD and Indiana DOT have the research category for the
research by special requests.

Table 1. Research categories provided by responded state DOTs

State DOTs Research Categories

Arizona Materials, Structures, Roadway, Construction, and Maintenance | Traffic, Safety,
Intelligent Transportation Systems, and Transportation System Management and
Operations | Planning, Policy, and Communication | Environmental
Colorado Structure, Geotechnical, and Hydraulics | Environmental and Water Quality | Material
and Pavement | Safety, Maintenance, and Planning
Florida Construction | Environmental Management | Geotechnical | Maintenance | Materials |
Planning | Public Transportation | Roadway Design | Safety | Structures | Traffic
Engineering and Operations/ITS | Turnpike
Georgia Mobility | Asset Management | Safety | Work Force & Policy Development
Illinois Construction | Environment | Pavement Design, Management, and Materials |
Planning | Public and Intermodal Transportation | Safety Engineering | Structures,
Hydraulics, Geotechnical | Sustainability | Traffic Operations, Roadside Maintenance
Indiana Pavement | Materials | Construction | Geotechnical | Technical Services | Safety |
Highway Maintenance | Special requests from the Executive staff | Traffic | Financial
and Asset Management
Iowa Safety | Pavement | Maintenance | Bridge and Structure | Human Factors | Intelligent
Compaction/Construction
Louisiana Pavements | Asphalt Materials | Concrete Materials | Geotechnical | Structures | ITS |
Safety | Special Studies
Minnesota Materials & Construction | Traffic & Safety | Maintenance Operations | Bridges &
Structures | Policy & Planning | Multimodal | Environmental | Administration |
Federal Program Support
Montana Bridge and Hydraulics | Environment | Highways | Maintenance | Planning | Traffic |
Safety
New Hampshire Aeronautics | Bridges | Construction | Environment | Geotechnical | Maintenance |
Materials | Pavements | Traffic | Transit
North Carolina Planning/Programming, Policy/Transit | Environment & Hydraulics | Pavement,
Maintenance, & Materials | Traffic, Safety, & Roadway Design | Structures,
Construction, & Geotechnical
Ohio Administration | Aerial | Construction | Environmental | Geotechnical | Hydraulic |
Maintenance | Materials | Pavements | Planning | Policy Development | Roadway |
Safety | Structures | Traffic
Oregon Structures | Maintenance & Operations | Active and Sustainable Transportation |
Traffic Safety/Human Factors | Construction, Pavements, & Materials |
Planning/Economic Analysis | Geotechnical, Hydraulics & Environmental
Texas Construction & Maintenance | Safety & Operations | Structures & Hydraulics |
Planning & Environmental | Strategic Planning
Utah Materials | Maintenance | Structures | Geotechnical | Traffic Management and Safety |
Planning | Asset Management | Preconstruction | Environmental and Hydraulics |
Transportation Innovation | Public Transportation
Washington Traffic Operations | ITS/Congestion | Maintenance Operations | Freight Mobility |
Passenger Rail | Tolling | Innovative Partnerships | Emergency Management |
Environment | Planning | Bicycle/Pedestrian | Aviation | Public Transportation | Noise
| Air Quality | Hydraulics & Storm Water | Pavements & Materials | Construction |

— 48 —
 State DOTs Research Categories
Practical Solutions | Multimodal Planning | Highway Geometric Design | Design
Safety | Utilities | Right of Way (ROW) | Real Estate | Roadside Features &
Landscape | Computer Aided Engineering (CAE)
West Virginia Safety/Traffic | ITS/GIS | Pavements | Geotechnical | Freight | Structures |
Maintenance | Construction | Environmental | Materials | Legal/Personnel/Civil
Rights | Transportation Finance | Planning/Transportation/Air Quality

Question 2: Have you ever developed any quantification process to measure and
document the value of completed research?

Question 3: Would you please provide the link or attach the related documents?

Questions 2 and 3 are related. Out of 27 respondents to Survey-1, 14 state DOTs stated
that they developed their quantification processes to determine the value of transportation
research results in Question 2. The rest of the state DOTs answered “No” as shown in
Figure 5. In addition, 11 state DOTs provided links where the relevant documents can be
retrieved, or the state DOTs attached the documents (see Table 2). The document of the
Arizona DOT is an implementation report which provides information regarding the
extent to which research projects have been implemented or are inactive. The documents
obtained from the Colorado and Wyoming DOTs present the resultant impacts of the
research projects, but there are no examples for the quantification of research values.
Florida DOT provided a survey link for determining the value of research. The survey
was created by project managers to investigate the outcomes of the research projects. The
Utah and Texas DOTs presented their reports to determine the value of transportation
research. The West Virginia DOT provided a research manual that shows the research
categories and procedures to develop state-needed research projects. The Missouri,
Montana, and Indiana DOTs provided their reports that contain examples for the
quantification of research value. On the other hand, one state DOT, Oregon, provided
comments on how to assess research projects. They mentioned that completed projects
are assessed and identified as one of the four categories: changed agency practice,
validated current agency practice, project objectives were met but the result was
inconclusive, and others (e.g., failed projects).

— 49 —
 Figure 5. State DOTs that responded to question 2

Table 2. Summary of document and links submitted by the state DOTs

State Documents/Link

Arizona Implementation of Research at the Arizona Department of Transportation SPR‐727

Colorado Excel file (Implementation report)

- Survey link (https://www.surveymonkey.com/r/QN99X58?sm=pi g9usTI8%2f56Wl
ZAgmHC4g%3d%3d)
Florida - Other link not working (http://www.dot.state.fl.us/research-
center/Completed_Proj/Summary_PL/FDOT_BDK77_977-24_rpt.pdf)
- Quantification method: Determining the value of research for transportation in
Indiana
Indiana - An example of quantification of value: Use of Dynamic Cone Penetration and Clegg
Hammer Tests for Quality Control of Roadway Compaction and Construction -
SPR-3009)
- Example of quantification of value: The Value of Missouri Transportation Research-
Missouri Fiscal Year 2011
- The Value of Missouri Transportation Research-Fiscal Year 2012
- A link to an Example of quantification of value:
Montana http://www.mdt.mt.gov/other/webdata/external /research/docs/research_proj
/rest_area/pm_report.pdf

Oregon - They provided comments regarding a research project assessment

— 50 —
 State Documents/Link
- Excel file for calculation of benefit-cost ratio
Texas - A report on determining the value of research results
- Link to a guideline:
Utah https://www.udot.utah.gov/main/uconowner.gf?n=1339002847990478

West Virginia - Research manual

Wyoming - Self-evaluation report (Implementation report)

Figure 6. State DOTs for present/future plans to develop quantification processes

Question 4: Do you have any present/future plans to develop any quantification process
to determine the value of research benefits?

For this question, 11 state DOTs of 27 respondents stated that they have present/future
plans to develop a quantification process as shown in Figure 6. Also, eight state DOTs
(IA, IL, LA, MO, NH, NY, MD, and SC) answered “Not Sure” for any present/future
plans to develop quantification process at the time when the survey was conducted.

— 51 —
Question 5: May we follow up with you to get more information in the future?

This question was asked in case the research team requires a follow-up contact to get
more details based on the data analysis results of Survey-1. A total of 25 state DOTs
agreed to this question as shown in Figure 7.

Figure 7. State DOTs (including DC) who agreed to follow-up in future

Survey-2 Statistics

The list of the surveyees that was used for Survey-1 was also re-used to deliver Survey-2.
A total of 20 state DOTs, which represents the response rate of 40%, participated in the
survey. All these state DOTs are graphically displayed in Figure 8. The District of
Columbia DOT participated in this survey but is not displayed in the map due to its size.
The screenshots of Survey-2 questionnaire and corresponding responses can be found in
Appendices B1 and B2, respectively.

— 52 —
 Figure 8. State DOTs (including D.C.) participated in survey-2

Question 1: If your organization has research categories to develop research projects and
uses criteria to classify the research categories, please list the criteria in the textbox
below; otherwise, skip this question.

This question was initially designed to find common criteria for research categorization
among state DOTs. A total of 14 state DOTs answered this question as shown in Figure 9.
However, as shown in Table 3, some of the inputs provided by the state DOTs (GA, LA,
TX, VA, WA, and WI) are related to research categories or research benefits rather than
research categorization criteria. In addition, the New York DOT clearly mentioned that
they do not have any criteria used to categorize research projects. These inappropriate
answers left only seven state DOTs available for the data analysis. The Nevada and
Rhode Island DOTs consider the state goals and state needs, respectively. The California
and Illinois DOTs utilize advisory committees to develop research categories. The
Arizona and Indiana DOTs stated that they develop research categories based on the
department’s strategic focus areas and specialty areas, respectively. Lastly, the
Washington DC DOT considers the agency’s organizational structure. It appears that
there is no single common criterion used for most of the responding state DOTs to
categorize research. However, if the criteria provided by AZ, CA, IL, and IN are
established in the consideration of states’ goals or needs at the end, it would be possible

— 53 —
to conclude that the other six DOTs (AZ, CA, IL, IN, NV, and RI) categorize research
based on the strategic goal of their individual state to address its current/future needs.

Figure 9. States that Responded to Survey 2

Table 3. The information provided by state DOTs for question 1 of Survey-2

State DOT Answer

AZ Department's strategic focus areas
CA Research steering Committees
DC Agency organizational structure
GA Mobility, Safety, Policy / Workforce Development, and Asset Management
IL Technical Advisory Groups
IN Based on the specialty areas
Policy and Planning, Special Studies, Pavements (both asphalt and concrete are included
LA as well as our Accelerated loading program), Asphalt Materials, Concrete Materials,
Structures, ITS, Safety, and Geotech.
NV State goals
NY We do not
RI State needs
Construction and Maintenance Planning and Environmental Safety and Operations
TX
Structures and Hydraulics Strategy and Innovation
Bridge structures and materials, pavement structures and materials, safety, system
VA
operations and congestion mitigation, planning, environmental stewardship and

— 54 —
 State DOT Answer
sustainability, non-destructive testing and evaluation methodologies, data analytics to
better make informed decisions
WA Strategic, Enterprise risk, benefit, implementation
WI Flexible pavements, rigid pavements, structures and geotechnics

Question 2: If your organization has ever used benefit categories (e.g., improved safety,
improved mobility, etc.) to determine the value of transportation research results, please
list them in the box below; otherwise, skip this question.

Out of 20 state DOTs, nine state DOTs provided their benefit categories. The state DOTs
that responded are displayed in Figure 10. The inputs provided by the state DOTs that
responded are summarized in Table 4. The data analysis results based on the inputs in
Table 4 show that “Safety,” “Mobility,” “Infrastructure,” “Cost,” “Time,” “Knowledge,”
“Life-cycle,” and “Sustainability” connected to themes such as “Efficient,” “Reliable,”
“Reduced,” “Improved,” and “Increased,” which were the benefit categories most
frequently used by the responding state DOTs.

Figure 10. State DOTs responding to question 2 of survey 2

— 55 —
 Table 4. Benefit categories of the responding state DOTs

State Benefit Category

Planning for Zero Fatalities; Better Design of Roads and Bridges; Sustainable
Infrastructure and Maintenance; Efficient and Safer Traffic Operations; Efficient
Nevada
Multimodal Mobility; Better Project Delivery; Transformative Transportation
Technologies
Safety/Health; Stewardship and efficiency; Sustainability, Livability and Economy; System
California
performance; Organizational Excellence
Construction Savings; Decreases Engineering/Administration Costs; Decrease Lifecycle
Illinois Costs; Environmental Aspects; Impact on IDOT policy; Increase Lifecycle; Operations and
Maintenance Savings; Safety; Technology; User benefits
Indiana Cost savings; Safety; Mobility/reduced congestion; Quality; Time savings
Improved safety; Improved congestion/mobility; longer lasting pavements and structures;
Virginia Improved inspection/maintenance practices; Strategic health monitoring to improve
infrastructure assessment; Improved decisions stemming from data analytics
Improved mobility; Improved safety; longer service life; Improved efficiency (i.e. reduced
Louisiana man hours); Cost-benefit analysis; Life-cycle analysis (LCA); Life cycle cost analysis
Improved environmental conditions; Lower initial cost
Iowa Safety; Mobility; Sustainability; Technology
Cost savings; Time savings; Knowledge increase; Leverage (e.g., demonstrating
Ohio partnership/commitment/compliance); Process improvement; Level of Knowledge;
Management and Policy; Quality of Life; Customer Satisfaction
Environmental Sustainability; System Reliability; Increased Service Life; Improved
Productivity and Work Efficiency; Expedited Project Delivery; Reduced Administrative
Costs; Traffic and Congestion Reduction; Reduced User Cost; Reduced Construction,
Texas
Operations, and Maintenance Cost; Materials and Pavements; Infrastructure Condition;
Freight movement and Economic Vitality; Intelligent Transportation Systems; Engineering
Design Development/ Improvement; Safety

Question 3: If your organization has ever used qualitative/quantitative measures for each
of the benefit categories listed in Question 2, please list them in the box below; otherwise,
skip this question. (You can write up to three measures for each benefit category).

A total of six state DOTs provided their benefit measures in response to Question 3,
which are identified in Figure 11. The benefit measures provided by the responding state
DOTs are listed in Table 5. The attributes of these benefit measures are both quantitative
and qualitative. For example, California DOT quantitatively measures the number of
reduced worker/user fatalities and injuries for the benefit category “Safety/Health,”
which also considers the qualitative benefit measure “Promote community health through
active transportation.” However, it was observed that some of the benefit categories were
not accompanied by any benefit measure, as shown in Table 5.

— 56 —
 Figure 11. State DOTs that provided benefit measures

Table 5. Benefit measures given by state DOTs

State Benefit categories Measures

Zero worker fatalities; Reduce user
Fatalities(F) and injury(I); Delayed reporting
California Safety/Health
of F&I; Promote community health through
active transportation
Effectively manage transportation elements;
— Stewardship and efficiency Efficiently deliver projects and service on
time
Improve quality of life in CA; Reduce
Sustainability, Livability and
— Environmental impacts; Improve economic
Economy
prosperity
Improve travel time reliability; Reduce peak
— System performance period travel through ITS, strategies; Improve
integration and operation of system
Promote positive work environment; Increase
— Organizational Excellence customer satisfaction; Improve
communication
Illinois Construction Savings Materials; Labor; Time
Decreases Engineering/
— Planning/Design Costs; paperwork
Administration Costs

— 57 —
 State Benefit categories Measures
— Decrease Lifecycle Costs Ongoing maintenance costs
Pollution; Hazardous waste; Material
— Environmental Aspects
reduction/recycling
— Impact on IDOT policy
— Increase Lifecycle
— Operations and Maintenance Savings Materials; Labor/time; Equipment
— Safety Reduction of crash frequency; Lives saved
— Technology Technology transfer; New materials/methods
Time (traffic congestion); Money saved to
— User Benefits
tax payers
Benefit-cost ratio; Marginal Internal Rate of
Indiana Cost savings
Return
Severe crashes on state controlled roads;
— Safety
Impacts of research to safety
Travel time reliability; Impacts of research to
— Mobility/reduced congestion
reduce congestions
— Quality Impacts of research to quality of materials
Impact of research to time savings in testing
— Time savings
and construction
Louisiana Improved mobility Reduced travel time; Reduced congestion
Reduction in crashes; Reduction in lives lost:
— Improved safety
difficult to get all crash data consistently
— longer service life Improved life cycle cost; # years increased
Improved efficiency (i.e. reduced Reduced man hours to perform a task; Less
—
man hours) people required to complete a task
Cost of research/implementation vs.
— Cost-benefit analysis measured improvement (could be man hours,
increased service life, etc.)
— Life-cycle analysis (LCA) # years extension vs. status quo
# years extension vs status quo with initial
— Life cycle cost analysis
costs and maintenance costs included
— Improved environmental conditions Reduction in CO2 emissions
— Lower initial cost $ saved in construction; $ saved in design
Ohio Cost savings $ saved per unit (e.g., project, mile)
— Time savings # hours saved per unit (e.g., project, mile)
— knowledge increase Specific learned items from project
Leverage (e.g., demonstrating
—
partnership/commitment/compliance)
— Process improvement
Virginia Improved safety Number of lives saved; Reduction in crashes
Improved travel time through congested
— Improved congestion/mobility
corridors

— 58 —
 State Benefit categories Measures
Longer lasting pavements and Increased service life; Minimizing
—
structures maintenance costs
Improved inspection/maintenance
—
practices
Strategic health monitoring to
—
improve infrastructure assessment
Improved decisions stemming from
—
data analytics

Question 4: If your organization has ever experienced any challenges/constraints to

using the measures, please explain the challenges/constraints in the box below.

Out of 20 respondents, five state DOTs provided their challenges/constraints and are
indicated on the map in Figure 12. Table 6 shows the inputs of the state DOTs responding
to this question. The significant challenges/constraints identified, which included the lack
of data for newly-introduced benefit measures or which require long-term monitoring; the
lack of the resources to monitor long-term performance and collect the required data; and
the reliability of the collected data.

Figure 12. Map of state DOTs that provided challenges

— 59 —
 Table 6. Challenges/constraints to use the measures

State Measures Challenges/Constraints

California Zero worker fatalities —
Reduce user Fatalities(F) and injury(I) —
Promote community health through active
New area has lack of data
transportation
Effectively manage transportation elements —
Efficiently deliver projects and service on time —
Improve quality of life in CA New area has lack of data
Lack of consistent methods - Green
Reduce Environmental impacts
House gases
Improve economic prosperity New area has lack of data
Improve travel time reliability —
Reduce peak period travel through ITS, strategies New area has lack of data
Improve integration and operation of system New area has lack of data
Promote positive work environment —
Increase customer satisfaction —
Improve communication —
Indiana Benefit/cost ratio Monitoring the B/C in a long term
Lack of data for the cost of
Marginal Internal Rate of Return
implementation
Severe crashes on state controlled roads —
Impacts of research to safety —
Travel time reliability —
Impacts of research to reduce congestions —
Lack of resources to monitor the
Impacts of research to quality of materials
benefits in the long term
Impact of research to time savings in testing and Lack of resources to monitor the
construction benefits in the long term
Louisiana Reduced travel time —
Reduced congestion —
Reduction in crashes —
Difficult to get all crash data
Reduction in lives lost
consistently
Very difficult to account for all
Improved life cycle cost
factors objectively
# years can be difficult to estimate,
# years increased especially if it is on the 100 year
timeframe
Reduced man hours to perform a task —
Less people required to complete a task —

— 60 —
 State Measures Challenges/Constraints
Cost of research/implementation vs. measured
—
improvement
# years can be difficult to estimate,
# years extension vs. status quo especially if it is on the 100 year
timeframe
# years can be difficult to estimate,
# years extension vs status quo with initial costs
especially if it is on the 100 year
and maintenance costs included
timeframe
Reduction in CO2 emissions —
$ saved in construction —
$ saved in design —
Personnel uncertain of estimating
Ohio $ saved per unit (e.g., project, mile)
savings rate or number of units
Personnel uncertain of estimating
# hours saved per unit (e.g., project, mile)
savings rate or number of units
Specific learned items from project Lack of consistent methods
Virginia Number of lives saved —
Reduction in crashes Incomplete crash data
Improved travel time through congested corridors —
Increased service life —
Capturing all maintenance costs
Minimizing maintenance costs
accurately

Research Categories and Subcategories

Based on the information collected in the literature and discovery search, the research
areas in the nonhierarchical structure were identified. Then, the project categorization
method proposed by Archibald (2013) was used to define the mutually exclusive higher-
level categories. This method consisted of two components. First, the purpose of the
project categorization: Archibald defined two main strategic purposes: (a) project
selection - determining which potential projects are to be funded and executed; and (b)
prioritization of selected projects - determining the relative importance of selected
projects to assist in allocating scarce resources. Second, project attributes (criteria)
selection - refers to project characteristics, such as application area, geography, and
complexity. Crawford et al. (2005) concluded that “all organizations that have large
numbers of projects must categorize them, although the categories are not always
immediately visible.” Also, they stated that “The categorization of projects is beneficial
and useful to organizations, but it needs to be practical and not theoretically oriented.” In

— 61 —
this study, it was apparent that it was important to define the goal of this research project,
which was to determine the value of transportation research results. Therefore, the
research team made every effort to identify criteria for this research that aligned with its
goal.

Furthermore, it was critical to defining criteria that leads to research categories with less
overlap, which covers most of the research projects sponsored by the state DOTs. Based
on the knowledge obtained from the literature review, the survey data analysis results, the
purposes of state DOT research programs, and the partnerships with state universities for
the research programs, the research team identified three different criteria: 1) state DOTs’
responsibilities (Type A), 2) civil engineering discipline areas (Type B), and 3) impact
assessment elements (Type C). Once the research categories based on these three types
were established, the subcategories of each of the research categories were determined to
increase the level of detail. Also, the coverage rates for the three types were investigated
by comparing the suggested research categories with the research categories used by state
DOTs.

State DOTs Responsibilities (Type A)

The primary goal of state DOTs is to provide safe, reliable transportation systems to
improve the quality of life of local residents and communities. The safety and reliability
of transportation systems should be secured during their entire lifetimes from planning
through operation and maintenance. In addition, it is becoming more critical as
transportation systems are sustainable for the environment and the economy for future
generations, which adds more responsibilities to the public agencies responsible for
transportation systems. To fulfill these responsibilities effectively and efficiently, state
DOTs operate research programs that develop research to find solutions to overcome
current and future challenges to build safe, reliable, and sustainable transportation
systems. Therefore, the state DOTs’ responsibilities include design, planning,
construction, operation, maintenance, safety, and sustainability. To develop the
subcategories for each of the candidate research categories based on the state DOTs’
responsibilities, the properties of the categories were further investigated. As a result, the
identified research categories and subcategories based on Type A are shown in Figure 13.

— 62 —
 Figure 13. Research categories and subcategories based on type A

Civil Engineering Discipline Areas (Type B)

The development of the research categories based on Type B was based on the
understanding that state DOTs contract state universities, especially the departments of
civil engineering, to conduct their research. Therefore, the accepted civil engineering
discipline areas were considered as the basis for the development of the Type B research
categories. For this purpose, various civil engineering research programs at U.S.
universities were investigated, which included structural engineering, geotechnical
engineering, transportation engineering, construction engineering and management,
hydraulic engineering, material engineering, and environmental engineering. The
characteristics of each of the discipline areas were further investigated to develop the
following research subcategories:

— 63 —
Structural Engineering: Design of bridges, buildings, and other types of structures using
concrete, steel, wood, masonry, and composites. It should be noted that the main focus of
most state DOTs is bridge structures.

• Geotechnical Engineering: Design of pavement, foundation, and earth

retaining walls and the study of rock and soil mechanics to investigate
subsurface and geologic conditions.
• Transportation Engineering: Application of technology, scientific
principles, and integrated strategies focused on smart and sustainable
development in planning, functional design, and operation and management of
facilities for any mode of transportation for movement of people and goods.
• Construction Engineering and Management: Management of construction
sites and processes focusing on project management, process modeling and
simulation, service-life prediction and life-cycle costing, and information
technologies.
• Hydraulic Engineering: Management of urban water supply, the design of
urban storm-sewer systems, and flood forecasting.
• Material Engineering: Design, manufacture, and characterization of
engineering materials for specific applications, focusing on asphaltic and
concrete pavements, recycled construction materials, pavement evaluation
corrosion within structures, nondestructive testing, and fracture and damage
mechanics.
• Environmental Engineering: Treatment of chemical, biological, and thermal
wastes, purification of water and air, and remediation of contaminated sites
after waste disposal or accidental contamination.

Based on the identified discipline areas and their characteristics, the research categories
and subcategories were developed as shown in Figure 14.

— 64 —
 Figure 14. Research categories and subcategories based on type B

Impact Assessment Elements (Type C)

In general, transportation systems affect communities in a variety of ways. For example,

they can help revitalize business districts, stimulate economic development, improve
access to jobs, and reinforce growth management. The process to evaluate the effects of
transportation systems on a community and its quality of life is called a “community
impact assessment” (Kramer and Williams, 2000). The assessment measures the extent to
which transportation systems address a variety of important topics, such as environmental
impacts including: air/noise quality and water quality; social impacts which are
categorized as mobility, safety, and design considerations; as well as economic impacts as
transportation projects tend to affect businesses. Potential economic impacts on
businesses include changes in business activities and land use due to the need for right of
way. Furthermore, changes in economic activities may influence the demand for freight
services. Since state DOTs solicit research topics within these areas, the impact
assessment elements were considered as a basis for the development of the Type C

— 65 —
research categories. Figure 15 presents the Type C research categories based on the
impact assessment elements.

Figure 15. Research categories and subcategories based on type C

Estimation of Coverage Rates

Once research categories and subcategories have been developed based on the three
different types, the coverage rates at the subcategory-level for each type were estimated
by investigating how many numbers of the research categories used in any state DOT can
be explained by the suggested research subcategories. As a result, the credibility of the
suggested research categories can be tested by answering the following two questions:
how distinct is each category from the other categories? Also, how inclusive is the
hierarchical category structure for the maximum coverage of the research areas
developed among the state DOTs? For the coverage rate, a simple formula was used as
shown in equation (1), where CR: coverage rate, a: the number of the research categories
of a state DOT covered by the proposed research subcategories, and b: the total number
of the research categories of a state DOT:

𝐶𝑅 = 𝑎 ⁄ 𝑏 × 100(%) (1)

— 66 —
Figure 16 shows an example to estimate the coverage rate (CR) based on equation (1).
The example considered the eight research categories used by ∆∆ DOT and the research
categories/subcategories based on Type A. The comparison was made at the level of the
Type A research subcategories. All research categories of the ∆∆ DOT except for “Special
Studies” are linked to the research subcategories Type A. It indicates that the coverage
rate of Type A for the ∆∆ DOT is 87.5% (= (7/8) × 100%). Similarly, the same procedures
were conducted to estimate all state DOTs’ coverage rates for Type A, B, and C research
categories developed in this project. Figure 17 represents the coverage rates of the three
different research category types for the state DOTs. Figure 18 compares the average
coverage rates of the three different types, which shows that Type A has the highest
coverage rate followed by Type B and Type C.

Figure 16. Example to estimate a coverage rate

— 67 —
 Figure 17. Coverage rates of type A, B, and C for all state DOTs

Figure 18. Average coverage rates of three different category types

— 68 —
 Benefit Categories/Subcategories and Measures

To develop the benefit categories and subcategories, the information from the two data
sources were utilized. The two data sources include the keywords captured from the state
DOTs’ mission statements and the benefit categories provided by the state DOTs’
responses to Survey-2. The state DOTs’ mission statements were available on their
websites. Once the mission statements were retrieved, the keywords of the mission
statements were identified through the thematic analysis. The captured keywords are
summarized in Table 7. In addition, a data processing step was conducted for the
keywords of the state DOTs who have already provided the benefit categories (see Table
4) in Survey-2.

Table 7. Benefit areas captured from state DOTs’ mission statements

State Result of Thematic Analysis

Alabama Safety, Efficiency, Sound Environment, Economic and Social Development
Alaska Quality of Life, Safety, Environmentally Sound, Reliability
Arizona Safety, Workforce Development, Infrastructure Health, Innovation, Financial Resources
Safety, Efficiency, Customer Satisfaction, Social Enhancement, Economic Development,
Arkansas
Environmental
Safety/Security, Infrastructure Preservation, Mobility, Economic Development,
California
Technology/Innovation, Environmental
Colorado Mobility and Accessibility, Environmental
Connecticut Quality of Life, Safety, Environment
Delaware Mobility and Accessibility, Safety, Environment, Partnership
Florida Infrastructure Health, Safety, Economic Development, Organizational Excellence
Georgia Safety, Environment, Economic Development, Partnership
Hawaii Safety, Efficiency, Mobility and Accessibility, Economic Development, Quality of Life
Idaho Safety, Mobility, Economic Developments, Innovation, Workforce Development
Kentucky Safety, Efficiency, Environment, Quality of Life
Louisiana Quality of Life, Economic Development
Maine Safety, Economic Development
Safety, Infrastructure Health, Quality of Life, Mobility and Accessibility, Project
Maryland
Delivery, Economic Development
Massachusetts Safety, Reliability, Economic Development, Quality of Life
Michigan Mobility, Efficiency, Partnership, Safety, Environment
Minnesota Safety, Accessibility, Reliability, Efficiency,
Mississippi Safety, Efficiency, Environment
Missouri Safety, Infrastructure Health, Reliability, Economic Development
Montana Quality of Life, Safety, Efficiency, Environment, Economic Development

— 69 —
 State Result of Thematic Analysis
Safety, Economic Development, Environment, Project Delivery, Infrastructure Health,
Nebraska
Partnership, Workforce Development
Nevada Safety, Innovation, Efficiency, Infrastructure Health
New Safety/Security, Customer Satisfaction, Mobility, Efficiency, Infrastructure Health,
Hampshire Partnership, Workforce Development
New Jersey Quality of Life
New Mexico Safety, Efficiency, Economic Development, Environment
New York Safety, Mobility, Efficiency, Economic Development, Environment
North
Safety, Customer Satisfaction, Efficiency, Reliability, Economic Development
Carolina
North Dakota Safety, Mobility
Efficiency, Mobility, Accessibility, Safety, Environment, Constructability, Cost
Ohio
Effectiveness
Oklahoma Safety, Economic Development,
Oregon Safety, Economic Development, Environment
Pennsylvania Sustainability
Rhode Island Safety, Efficiency, Environment, Mobility, Economic Development
South
Infrastructure Health
Carolina
South Dakota Safety, Efficiency
Tennessee Safety, Reliability, Economic Development
Texas Safety, Mobility
U.S. DOT Mobility And Accessibility, Quality of Life
Utah Safety, Infrastructure Health, Mobility
Safety, Infrastructure Health, Environment, Customer Satisfaction, Workforce
Vermont
Development
Virginia Safety, Mobility, Economic Development, Quality of Life
Washington Safety, Reliability, Economic Development
West Virginia Safety, Infrastructure Health, Economic Development, Environment
Wisconsin Safety, Efficiency
Wyoming Safety, Infrastructure Health, Efficiency, Environment

— 70 —
Figure 19. Proportions of keywords (Yoon and Dai 2017)

— 71 —
 Figure 20. Co-occurrence network of the keywords (Yoon and Dai 2017)

The keywords were analyzed for two purposes: identification of the frequent keywords
for the benefits and investigation of the keywords in co-occurrence for the benefit
categories and subcategories. Figure 19 shows the proportions of keywords used through
the analysis of missions statements and the benefit categories from Survey-2, where
“Safety” has the highest frequency of 21.2%, and it was followed by “Economic
Development” (12.7%), “Environment” (11.1%), and “Others” (7.9%) that includes:
“Project Delivery”, “Social Development”, “Cost-Effectiveness”, and so on. The
community structures were utilized for the decomposed words in the co-occurrence
network. The community structures organize the keywords in network nodes based on the
strength of their connections in order to find the nodes within the same community
(Newman and Girvan 2004). Figure 20 shows the result of the co-occurrence network
analysis. In Figure 20, the larger circles represent a higher frequency of the word being
used while the thicker lines represent a stronger co-occurrence. The dashed lines indicate

— 72 —
an external connection between two words in different communities. The circle in a white
background (“Benefits”) in Figure 20 represents that the word is independent, as it does
not belong to any other communities. Therefore, the co-occurrence network result can be
interpreted to develop all feasible benefit categories and subcategories as listed in Table
8.

Table 8. Feasible benefit categories and subcategories

Benefit Categories Benefit Subcategories

Improved Safety and Security
Improved Mobility and Accessibility Reduced Congestion
Increased Speed
Improved Travel Time
Travel Time Reliability
Reduced Travel Cost
Improved accessibility to facilities and services
Improved Infrastructure Capacity and availability
Asset condition
Remaining Life/Structural Capacity
Increased Lifecycle
Improved Environment Vehicle Emissions
Length or Extent of Air Quality Problem
Water Quality, Wetlands, Aquatic Life
Energy Impacts
Noise Impacts
Improved Technology/Innovation
Workforce Development
Expedited Project Delivery
Increased Knowledge
Improved Quality of Life
Customer Satisfaction

The determination of research values depends highly on the ability to evaluate the impact
of research outcomes through benefit measures. Mission statements generally describe
goals that organizations try to accomplish. They are also used to assist in establishing
strategic initiatives and monitoring organizational performance for successful businesses.
Therefore, to obtain and develop benefit measures, the mission statements of the 50 state
DOTs in the nation also was carefully examined. As a result, the possible benefit
measures corresponding to the benefit categories and subcategories in Table 8 were
captured as listed in Table 9.

— 73 —
 Table 9. Benefit measures corresponding to benefit categories/subcategories

Benefit Category Benefit Subcategory Measures

- Number of crashes by type, location type, and so forth
- # lives saved
- Dollar savings due to reduction in crashes
- Number (or rate per capita or number of travelers) of
Improved Safety crimes at rest areas, bus stops, highways, and so forth by
and Security type or severity
- Value of losses from theft per capita, person-trip, shipment
value, ton
- Reduction in number of incidents in construction sites
- Level of service (LOS)—measure of congestion from A–F
based on volume-to capacity ratio (facility-specific
measure)
Improved - Number of intersections congested (e.g., with LOS E or F)
Mobility and Reduced Congestion during peak hours
Accessibility - Travel time under congested conditions
- Lane-mile duration index (number of congested lane-miles
times the duration of congestion)
- Average speed for given roadway segment or origin-
Improved Speed destination pair
- Average travel time (by mode or cross modes) for a given
origin-destination pair or trip type
Improved Travel - Travel time from freight intermodal facilities to highway
Time facilities

Travel Time - Percent on-time shipments (by commodity or mode)

Reliability
- Trip cost by mode for origin-destination pairs
- Dollar losses due to freight delays
Travel Cost - Dollar savings
- Percent of urban population with convenient access to
public transit (e.g., living within a quarter mile of a transit
Accessibility to stop with a non-rush-hour service frequency of 15 minutes
Facilities and or less)
Services - Access time to passenger or intermodal facilities

Improved Capacity and - Intermodal terminal capacity

Infrastructure Availability
- Average health index (0–100 scale)
Asset Condition - Percent structurally deficient (SD)
- Number of steel bridges with section loss in a member
- Percent asset quantity out of service due to deteriorated
condition
Remaining - Reduction in distance (or time) between failures for transit
Life/Structural vehicles
Capacity - Age of fleet by vehicle type or remaining useful life for
vehicles

Increased Lifecycle - Dollar savings due to use of durable materials

- Vehicular emissions by type—NOx, VOC, CO2, CO,
ozone, fine particulate matter (PM2.5) —can be limited to
Improved Reduced Vehicle nonattainment areas and identified by source (e.g.,
Environmental Emissions passenger versus freight)

— 74 —
 Benefit Category Benefit Subcategory Measures
- Number of days that pollution standard index is in the
Length or Extent of unhealthful range
Air Quality Problem
- Acres of wetlands replaced or protected for every acre
affected by highway projects
- Level of fish habitat reduction as a result of new
Water Quality, construction
Improved - The amount of water leaving or discharging from the
Wetlands, Aquatic
Environmental system
Life, Farmlands
- Changes in open space, gardens, parks, farmlands and
wildlife habitat (#acres)
- Percent of vehicles using alternative fuels
Energy Impacts - Average fuel consumption
- Dollar savings
- Number of residences or percent of population exposed to
highway noise exceeding established standards (or greater
Noise Impacts than X decibels)
- Number of noise receptor sites above threshold
- Percent of contracts (or contract value) completed on-time
New Technology/ - Percent of contracts (or contract value) completed on-
budget
Innovation - Reduction in emission

Workforce - Rating the effectiveness of the workforce training program

Development
- Percent of contracts (or contract value) completed on-time
- Percent of contracts (or contract value) completed on-
Expedited Project budget
Delivery - Number of contractor partnerships
- Reduced installation time (# days, #hrs., etc.)
- Rating the project based on the following criteria:
a. This project expands the DOT knowledge base.
b. This project expands the State knowledge base.
c. This project expands the National knowledge base.
Increased d. This project lays the foundation for future research.
Knowledge - Number of research projects/products improving the body
of knowledge in a specific area(s)) or decision-making
processes.
- Number of sponsored students
- Rating the project based on the following criteria:
a. This project will increase the psychological comfort of
users.
b. This project will produce an aesthetic improvement.
Improved Quality c. This project will improve transportation accessibility
of Life d. This project will improve the environment.
- Number of research projects/products improving or
protecting the natural environment
- Dollar savings by offering free services to customers
Customer - Customer satisfaction rating
Satisfaction

The benefit categories in Table 8 were further divided into the relevant benefit
subcategories for the measures. As the last step, the measures assigned to each of the
benefit categories and subcategories then were mapped to the research categories and
subcategories based on Type A which was finally selected by the research TAC. The

— 75 —
mapping table is presented in Appendix C. Some of the data types in the mapping table
are numbered as they can be interpreted based on the associated research categories and
subcategories (see Appendix F).

Gap Analysis

The gap analysis was conducted for the research categories and subcategories based on
Type A, which was finally selected for this research. Therefore, as the first step, a
comparison table that included the research categories and subcategories based on Type A
and the benefit measures associated with the research subcategories was created. Also,
the comparison table shows the research subcategories of the state DOTs which have
developed quantification procedures. Then, each of the benefit measures was justified by
whether there was at least one state DOT having a developed quantification process for
the measure in the second step. Finally, the research subcategories considered for the
comparison were identified as belonging to one of the three groups as follows:

• Well-established research subcategories for the benefit measures with more

than 80% quantification procedures
• Incomplete research subcategories for the benefit measures with the
quantification procedures between 80% and 33%
• Under-developed research subcategories for the benefit measures with less
than 33% quantification procedures

The data from 7 state DOTs in the STC (e.g., FL, GA, LA, MS, NC, VA, and WV) were
available for the gap analysis. Table 10 is a list of the complete, incomplete, and under-
developed research subcategories as a result of the gap analysis. The detailed analysis
results are shown in Appendix D. For example, the research category
“Pavement/Material/Geotech,” has four measures such as remaining life, reduction of
life-cycle cost, monetary savings, and volume of information sharing. All of the benefit
measures except for the last one have the quantification procedures, which indicates 75%
so that the research subcategory was identified as incomplete. On the other hand, the
research subcategory “Road Design” was grouped into a well-established one as the
quantification procedures for 4 of 5 benefit measures were developed by any state DOT.

— 76 —
 Table 10. A List of complete, incomplete, and under-developed research subcategories

Well-Established Incomplete Under-Developed

- Road Design - Pavement/Material/Geotech - Administration/Workforce
- Hydraulics - Bridge/Structure - Transport Security
- Project Management/ - Product Development - Society
Construction Methods - Traffic/Roadway
- Roadway - Public Transportation
- Transit - Economy
- Transport Safety
- Environment

— 77 —
 Utilization of the Mapping Table

The purpose of this chapter is to guide transportation agencies as they navigate the
mapping table to effectively identify the relevant data type(s) needed to quantify the
value of completed research. This chapter consists of subchapters that explain how to
identify the research categories (RCs) and research subcategories (RSCs), the measures
going through the benefit categories and subcategories, and the data type(s). Figure 21
illustrates the steps to utilize the mapping table. The successive three steps also include
the feedback loop between Step-2 and Step-3, which is prepared to reevaluate the benefit
categories and subcategories for alternative data types. That is, by moving forward on the
mapping table, transportation agencies can identify the single or multiple data type(s)
commonly associated with the benefit category and subcategory. However, although the
benefit category and subcategory may be clearly determined, there may not be a single
data type to measure the value of the completed research in question for the following
possible reasons: 1) the attribute of the information used to determine the benefit
category and subcategory is qualitative, and 2) the data type relevant to the benefit
category and subcategory is not available to the transportation agency. This situation
requires going back to Step-2 and measuring the research using alternative data type(s) in
a different benefit category and subcategory. The steps to explore the mapping table are
discussed in the last subchapter considering an actual recently completed research
project.

Figure 21. Steps to utilize the mapping table

— 78 —
 Research Categories and Subcategories

Most state DOTs classify and develop their research programs based on the generally
known transportation research areas (e.g., structures, pavements, asphalt materials, etc.).
The RCs and RSCs in this guidebook were developed considering the maximum
inclusion of the research areas in use by the state DOTs of the STC. Table 11 shows the
RCs and RSCs associated with the research areas of the STC states, which enables the
agencies to quickly identify the appropriate RC/RSC in order to evaluate the value of
research upon completion using Table 11. Occasionally, the “perfect” RC and RSC
cannot be found in Table 11 as the coverage rating of the mapping table is 95%. For these
occasions, the agency first must clarify the specific objectives and parameters of the
research (e.g., scopes and outcomes expected) because good background information and
in-depth knowledge of the research ultimately helps to identify the “best” suitable RC
and RSC. It is noted that a single research project could have multiple (e.g., two or three)
“feasible” RCs and RSCs depending on the nature of the objectives and expected
outcomes of the research. Appendix E presents the general definitions of the RSCs under
the associated RCs to assist agencies in selecting the best or most feasible categories
when it is not possible to identify the perfect RC/RSC in the mapping table.

Table 11. Research areas by STC

Research Research Subcategory Research Areas of the STC States

Category
Design Pavements/Material/Geotech Geotechnical | Pavement coating | Asphalt
materials | Soils
Bridge/Structure Beam design | Span lengths | Load studies
Road Curve studies | Interstate modeling
Hydraulics Drainage
Construction Project Management/ Project scheduling | Resource leveling | Resource
Planning Construction Methods allocation
Product Development Project progress plans
Planning Traffic/Roadway Intermodal planning | Freight transportation
Public Transportation Geographic Information System (GIS)
Administration/Workforce Policy development | Financial analysis | Business
operation
Maintenance/ Roadway Intelligence Transportation System (ITS) |
Operation Mobility | Legality | Civil rights | Maintenance
Transit Vehical maintenance | Transit route studies
Safety and Security Transport Safety Crash data analysis | Non - user impacts
Transport Security Quick response mobility | security

— 79 —
 Research Research Subcategory Research Areas of the STC States
Category
Sustainability Environment Environmental management | Environmental
analysis | Air quality
Energy

Data Type(s) through the Intermediate Steps

The main purpose for identifying the benefit category (BC) and the benefit subcategory
(BSC) is to help agencies with the transition from the RC/RSC to the measures and
subsequently the data type in the mapping table. Once the correct RC/RSC are chosen,
the subsequent step is to move on to the columns for the BC and BSC in the mapping
table. Each column provides minimal information, mainly key words that can be found in
the objectives and expected outcomes (or deliverables) of any research project. The
Measures column in the mapping table contains valuable information to help determine
the appropriate data type(s) needed. The Definition column further elaborates on the
meaning and definition of the measures in the Measures column. The major factor in
determining the correct RC/RSC is based on the objectives of the project. If a project
aims to achieve certain objectives, then it would be wise to choose a RC/RSC that has a
data type that is available.

The RSC largely relates to the general objective of the project and its final goal. For
example, if a project is within the Design category and is focused on the design of a
bridge, then the appropriate RSC would be Bridge/Structure (see Figure 22). The next
transition is from a RSC to a BC. This transition focuses on the benefit provided if the
main objective is accomplished. Using the same above example, a bridge can be
constructed for various reasons; and once again, based on the project objectives, an
agency can choose a RSC to a BC. This transition focuses on the benefit provided if the
main objective is accomplished. Using the same above example, a bridge can be
constructed for various reasons; and once again, based on the project objectives, an
agency can choose either “improved infrastructure” or “increased knowledge” on the
mapping table. After the BC is determined, the next transition is to the BSC, which
examines the ways to quantify the benefit category that was chosen. The BSC can be
broken down into categories such as lifecycle, economic, and knowledge. These choices
are largely up to the agency depending on their need. Assuming the largest factor the
agency wants to consider is the life of the product, they would be encouraged to choose
the economic option in BSC. Moving on from the BSC to the Measures is self-

— 80 —
explanatory and straightforward as all the measures are accompanied by definitions
(Appendix G) to assist in the decision-making process. Finally, the agency identifies the
data type(s) needed to determine the value of the completed research. Note that the
agency can use either single or multiple data type(s) considering the data availability, data
accessibility, and any other needs.

Example: Navigating the Mapping Table

The example research project used to illustrate how to navigate the mapping table is
“Categorization of Erosion Control Matting for Slope Applications,” which was
conducted by the Georgia DOT (GDOT). The research was developed to address how to
control the negative impacts that can result from high sediment loads in natural
waterways, focusing on different types of erosion control products that are available and
the best performers. Before the research began, GDOT had one category of erosion
control products and all the products were assumed to be interchangeable in their
performance. The ensuing research detailed the different considerations that would affect
the performance of erosion control devices, all of which primarily dealt with the
steepness of the slope to which the device would be applied. The next consideration was
the time required for new vegetation to regrow after a construction project was
completed. The final consideration on which the project focused was the lifespan of the
erosion control device based on the device’s material. This research was a follow up to a
previous research conducted by GDOT that reviewed the erosion control products used in
slopes.

— 81 —
Figure 22. Capture of the Mapping Table for the Design

— 82 —
Based on the previous research, various types of materials were examined to determine
the behavior of the material under different conditions.

Step-1: Determination of RC and RSC

Step-1 begins by determining the research area for which GDOT developed this research
so that the RC and RSC that match the research area can be identified from Table 11. The
agency then moves to the subsequent steps of determining the information for the BC,
BSC, Measures, and Data Type columns in the mapping table. If the research area is not
included in the mapping table, the information for this step likely would be in the
objectives and expected outcomes of their research in its final report. The report states
that the research was developed to find the following: 1) a design for erosion control on
steep slopes, 2) various practices that contractors can perform in regard to erosion
control, and 3) factors that affect the lifetimes of various erosion control devices. From
this information collected from the report, two main RCs (Design and Construction
Engineering) from the first two objectives were clearly recognized. However, the third
objective was somewhat vague for determining a relevant RC. In this situation, the best
way to find a RC is to refer to all possible RSCs in the mapping table and the general
definitions of them provided in Appendix E. Construction Engineering includes Project
Management/Construction Methods as a RSC which is defined as “….advanced
construction materials, equipment, technologies, and instruments…”. Therefore, the RC
considered for the last objective is Construction Engineering. Figure 23 describes the
three RCs for this research.

Figure 23. Determination of research categories

— 83 —
Using the RCs as a starting point, the report was further examined for more details about
the objectives and expected outcomes of the research to identify the RSC for each RC
determined. This action was made for the first two RCs as the last RC already indicated
the associated RSC. The report states that the steepness of slopes relates to the hydraulic
stress of the soil, which identified Hydraulics as the RSC for Design. The report also
emphasizes the way that the contractors implementing erosion control are managed as an
influence on the quality of the practices, which indicated Project
Management/Construction Methods as a RSC. Figure 24 is a partial capture of the
mapping table to show the RCs and RSCs determined for the example research. Figure 24
includes the columns for BC and BSC intentionally to show how the RCs and RSCs are
connected to the relevant BCs and BSCs in the next step.

Figure 24. RCs and RSCs determined for the example research

— 84 —
Step-2: Determination of BC and BSC

The determination of the BCs and BSCs for this project also utilized the information in
the report focusing on the research objectives and expected outcomes. The research was
developed to control the negative impacts on natural waterways from high sediment loads
by applying different types of enhanced-lifespan erosion control devices. Also, one of the
research deliverables includes the identification of the factors affecting the performance
of contractors on various erosion control practices. The research objectives and expected
outcomes indicated that the BCs and BSCs for this example research were: Improved
Environment and Water Resources/Flood Protection as the BC and BSC for Design –
Hydraulics; as well as Improved Project/Program Delivery and Accomplishment:
Effectiveness, Efficiency, Quality as the BC and BSC for Construction Engineering –
Project Management/Construction Methods (see Figure 24).

Step-3: Identification of Measure and Data Type

From the BCs and BSCs determined, the measures and data types for this research were
identified from the mapping table. The first data type was “Data of amount of water
discharge” for Improved Environment-Water Resources/Flood Protection in Design –
Hydraulics as shown in Figure 25. The data type identified was supported by the report of
this research, which states that a negative side effect of poor erosion control can cause
inadequate vegetation growth as a result of the water discharged through the erosion
control device. The report includes a table with the various amounts of water discharged
under different conditions.

Figure 25. Measure and data type for erosion control

— 85 —
 Figure 26. Data types under project efficiency measure

To identify the data type to measure the erosion control practices of contractors, the
possible data types of the three measures under the BSC—Accomplishment:
Effectiveness, Efficiency, Quality—were recognized as shown in Figure 26.

Figure 27. Measure and data type for contractors’ performance on erosion control practices

However, none of these data types were appropriate to directly quantify the performance
of the contractors on the erosion control practices because the attributes of the
information are qualitative. Therefore, it was necessary to investigate other possible data
types in different BCs and BSCs, but in the same RC and RSC initially determined
utilizing the report of the research to find a suitable data type. The enhanced performance

— 86 —
by the contractors can minimize water pollution in natural waterways. As a result, it
concluded that the performance quality of the contractors could be measured by the
alternative data type, “Ratio of restored and maintained area to disturbed area in
Improved Environment as a BC and Land Use” as a BSC as shown in Figure 27. The
third data type to measure the enhanced-lifespan erosion control devices was “Life-cycle
cost” for the measure Project Efficiency, as shown in Figure 28. The Project Efficiency
measure in the mapping table suggests two possible data types: “Duration of
construction” and “Life-cycle cost.” The latter data type can reasonably to be considered
as a measure for erosion control devices.

Figure 28. Measure and data type for enhanced-lifespan erosion control devices

— 87 —
 Evaluation Method to Determine Research Value

The research value measurement process basically assumes that: 1) the outcome of a
completed research can be obtained by measuring its immediate (< 2 years) or short-term
(2–5 years) performance based on the data type identified from the mapping table; 2) a
state DOT manages the historical outcome data, which is a dataset of the past research
using the same data type as the current research to be evaluated; and 3) a state DOT has a
target value for the data type. A target value is a long-term goal set by a state DOT to
ultimately achieve a benefit measure for the data type. However, it is not always the case
that the state DOT can collect historical data and set a target value because the data type
of research may not be identifiable or available. In particular, target values cannot be
determined for the data types pursuing maximum or minimum, such as maximizing cost
savings or minimizing life-cycle costs. It is also very challenging for the state DOTs to
provide an objective evaluation to qualitative research. Considering the state-of-the-
practice and challenges in determining the value of completed research, Table 12
represents all possible theoretical combinations which the state DOTs encounter in terms
of the existence of an outcome estimated, target value, and historical data.

Table 12. All possible combinations of target value, outcome, and historical data

Comb. Target Value Outcome Historical Data

1 O O O
2 O O X
3 O X O
4 O X X
5 X O O
6 X O X
7 X X O
8 X X X

In Table 12, Comb.-1 is the best condition to determine the value of completed research
while Comb.-8 is the worst. Comb.-3 and -7 might not be possible in practical terms
because it is more reasonable to assume that outcomes should be available if historical
data exist, which is a set of research outcomes accumulated. Comb.-2 and -6 might look
impossible because there should be historical data when the outcomes of completed
research are estimated. However, these cases could happen one time when the state DOTs
have just started to estimate and collect outcomes, which eventually will be moved to

— 88 —
Comb.-1 or -5. Comb.-4, which indicates that the state DOTs have set a target value to
determine the research value for a certain measure, but have no capabilities to collect the
data type for the measure currently. Therefore, this chapter provides information about
the state DOTs to use the measurement method and alternative approach for the research
under the following different condition:
• Case-1: research with a target value, an outcome measured, and historical data
• Case-2: research with a target value and an outcome measured, but no
historical data
• Case-3: research with a target value, but no outcome measured and no
historical data
• Case-4: research with an outcome measured and historical data, but no target
value
• Case-5: research with an outcome measured, but no target value and no
historical data
• Case-6: research with no target value, outcome measured, and historical data

The measurement method is a very straightforward and simple equation as shown in

equation (2). The equation is called Progress towards Target (PtT) in this guidebook as it
evaluates the progress of research outcomes to achieve the target value in percentage.

𝑃𝑡𝑇 = 𝑂𝑢𝑡𝑐𝑜𝑚𝑒/(𝑇𝑎𝑟𝑔𝑒𝑡 𝑉𝑎𝑙𝑢𝑒) × 100(%) (2)

The PtT equation primarily can be applied to quantify the value of the research in any
case, although it requires a subjective judgment for the cases with no outcome measured
(e.g., Case-3 and -6). However, using an alternative approach based on the statistical
confidence level can provide some extent of scientific basis to the subjective judgment of
transportation agencies. The cases with no target values should not be an issue when a
target value is always represented as 100% in a percentage. Also, a target value can be
newly set up or updated regularly when there is historical data available using the concept
of a Z-score (also called standard score). This becomes possible because a set of data
points that were collected in the past or will be collected in the future for the data type is
generally assumed to follow a normal distribution curve with a mean (μ) and standard
deviation (σ) of the dataset as shown in Figure 29.

— 89 —
 Figure 29. Normal distribution curve of data points

The normal distribution is converted to a standard normal distribution at which the mean
is 0 and the standard deviation is 1, using the Z-score equation (3), where x is the
outcome of a completed project to be evaluated for its value, μ is a mean, and σ is a
standard deviation. In the standard normal distribution, the Z-score is represented as the
number of standard deviations from the mean as shown in Figure 30.

𝑍 = (𝑥 − 𝜇)/𝜎 (3)

Figure 30. Standard normal distribution showing Z-scores

— 90 —
 Examples of the Possible Cases

To show how the measurement method in equation (1) and the alternative approach are
used for the six cases, a hypothetical example is presented in the following subchapters.
The hypothetical example revolves around an ongoing effort performed by a state DOT to
improve the average health index of various single span bridges across the state. It is
assumed that a research project was conducted to develop a new preservation material for
concrete overlays to extend their lifetimes and was completed in 2013. The RC chosen
for the research is Design, which leads to a RSC of Bridge/Structure in the mapping
table. The BC and BSC associated to the RC and RSC are Improved Infrastructure and
Lifecycle. Assuming the impact of the example research completed in 2013 on the
average health index would start presenting after five years, the data required to measure
the value of the research is the average health index of the bridge network in year 2018.

Case-1: Research with a Target Value, Outcome, and Historical Data

The DOT is in a perfect position to measure the value of complete research. Figure 31
shows the historical data of the average health index going back to 2004. The average and
standard deviations are the statistics of the historical data from 2004 to 2018. It is
assumed that the DOT has an established target value to achieve the average health index
of 90 percent on all their single span bridges in 2004. The PtT as the value of the outcome
of the example research completed is 95.0% (= (76⁄80) × 100%) using equation (2).

Figure 31. Historical data of average health index, research outcome, and target value

Since the target value was established in 2004 considering the average health indices at
that time, the state DOT may decide to revise the target value to address the average
health index trends in recent years. As a target value is the goal of the state DOT
represented as 100%, it can be done very easily by employing the concept of the Z-score
and the cumulative percentage in a standard normal distribution as shown in Figure 30.
That is, the Z-score of a target value is +4 standard deviation when the standard

— 91 —
deviations between -4 and +4 are used. As long as the historical data is available, the
statistics for the average and standard deviation can be obtained. By rearranging equation
(2) for x, the new target value that can be suggested to the state DOT is 87.8 (= (4×5.22)
+ 66.9). Therefore, the state DOT can rationally revise the target value considering the
value suggested and apply the revised one to compute the PtT of complete research
outcomes from 2019.

Case-2: Research with a Target Value and Outcome, but no Historical Data

The research in Case-2 considers that everything is kept constant as in Case-1 except for
the historical data and a different target value of 90 as shown in Figure 32. The different
target value is intended to show various PtT results. As there are no historical data, the
average and standard deviations are not available. The state DOT started collecting the
average health index of single-span bridges in 2018 to measure the five-year impact after
the example research was completed in 2013. The average health index collected was 76
for the target value of 90 set in 2018. The PtT for the outcome of this research is 84.4%
(= (76⁄90) × 100%) from equation (2). The research in Case-2 eventually moves to Case-
1 as the outcomes are accumulated in the following years.

Figure 32. Research outcome and target value, but no historical data

Case-3: Research with a Target Value, but No Outcome and Historical Data

The research in Case-3 is encountered under the condition that the state DOT has no
capability to monitor or control the data type identified from the mapping table to
determine the research value. In this case, the only way to handle this situation is to use a
subjective inference for the research outcome based on the confidence level. The
confidence level in statistics is equivalent to the cumulative percentage in Figure 30. In
this guidebook, the confidence level can be interpreted as how certain the transportation
agency is about the performance of the research towards a target value or how well the
transportation agency satisfies the research pace. That is, if the research was conducted at

— 92 —
an average pace, the confidence level may be assumed as 50% in terms of a cumulative
percentage on the standard normal distribution curve, so that the PtT is 50% (=
((50%)⁄(100%)×100%)). If the research proceeded at a better pace than expected, the
confidence level could be assumed to be higher than 50%.

Table 13. Tentative confidence levels for research performance or pace

Research Performance or Pace Confidence Level (CL)

Very Good 80% ≤ CL
Good 60% ≤ CL < 80%
Fair 40% ≤ CL < 60%
Poor 20% ≤ CL < 40%
Very Poor CL < 20%

To help the agency’s engineering judgment to determine the confidence level of a

research performance or pace, Table 13 is presented on a temporary basis until the state
DOT is capable of collecting the data type required for research outcome to move to
Case-2 and finally Case-1. For example, if the agency considers any research
performance or pace as a “Good,” the confidence level could be any percentage in the
range. Normally, the agency could simply choose the average of the range, which is 70%
for this example. However, a lower level could be considered, such as 60% or 65%, if the
agency is not confident in that range.

For the hypothetical example with a target value of 90 only as shown in Figure 33, the
state DOT is “fairly” but very “confident” that the average health index of the single-span
bridges has been improving recently, based on the visual inspection since they applied the
new preservation material as a result of the example resarch. Therefore, the agency
considers 60% as the confidence level of the reserch outcome for the target value of 90,
which is equivalent to 100% in a confidence level so that the PtT is 60.0% (= ((60%) ⁄
(100%)) × 100%).

Figure 33. Target value, but no research outcome and historical data

— 93 —
Case-4: Research with an Outcome Measured and Historical Data, but No Target
Value

In this case, the state DOT has no target value set up to determine the value of complete
research. Without the target value, it is impossible to use the PtT equation so a target
value must be developed. As mentioned earlier, a new target value can be developed
using the Z-score and cumulative percentage in Figure 30, having statistics of the
historical data. In Figure 34, the average and standard deviation of the historical data are
66.9 and 5.22, respectively. The Z-score of a target value is always +4 standard deviation
in a standard normal distribution between -4 and +4. Using equation (2), the target value
that can be suggested to the state DOT is 87.8 (= (4×5.22) + 66.9). When the target value
is determined at 88.0, the PtT is 86.4% (= (76.0⁄88.0) × 100%). It should be noted that the
target value is tentative based on the historical data, which implies that the state DOT
revises the target value at the time when the outcome of any research meets or exceeds
the target value.

Figure 34. Research outcome and historical data, but no target value

Case-5: Research with an Outcome Measured, but No Historical Data and Target
Value

The research using a data type that the state DOT has just started collecting and
monitoring or the research with data types that have no upper or lower boundaries (e.g.,
maximizing cost savings or minimizing annualized maintenance cost) falls into Case-5.
For the research in the former condition, the best approach for a state DOT is to set up a
new target value so that the outcome measured can be applied to the PtT equation in
[equation (1)]. However, if the state DOT is not in a position to do so, the most plausible
way to determine the value of the complete research relies on subjective engineering
judgment based on the confidence level suggested in Table 13. Similarly, the research in
the latter condition takes the outcome into account for the concept of the confidence
level. Figure 35 is a hypothetical example for Case-5. As the data type of this example is

— 94 —
an average health index, the tentative target value could be established at 90 out of 100 so
that the PtT is 84.4% (= (76.0 ⁄ (90.0) × 100%)). If determining the target value of the
research is ambiguous, the state DOT evaluates the performance or pace of the research
based on the confidence levels in Table 13. The confidence level selected becomes the
PtT as the target value is 100%.

Figure 35. Research outcome, but no historical data and target value

Case-6: Research with No Outcome Measured, Historical Data, and Target Value

The research in Case-6 is most challenging to the state DOT as there is no way to
determine the value of the research in an objective manner but instead relies on a
subjective judgment. Some studies suggested methods such as a survey and a focus
group. A focus group is a type of group decision in a small group to minimize the
subjectivity of the decision-making process. However, the state DOT might not be able to
conduct such methods and is looking for a simple approach, such as Table 13, which is
still subjective but mitigates the subjectivity issue to some extent. Figure 36 shows an
example for Case-6. The state DOT evaluates the performance or pace of the research
completed towards a target of 100% using the tentative confidence levels in Table 13.
The confidence level to be selected is a PtT value.

Figure 36. No research outcome, historical data, and target value

— 95 —
 Various Discrete Rating Systems

The values of the completed research outcomes in any of the cases are measured in terms
of percentages based on the PtT equation. These percentages can be classified into a few
different types of ratings or grades. Table 14 shows examples for the five-point rating or
grade for different scores in percentages. As shown in Table 15, a descriptive scale based
on percentages also can be considered. DOTs can select any form of measurement scale
that is suitable. However, it is recommended that the scale be consistent throughout all
state DOTs for uniformity purposes.

Table 14. Five-point ratings or grades for different scores

Rating % Score Grade

5 85% 90-100 93-100 A
4 70% 80-89 92-85 B
3 50% 70-79 77-84 C
2 30% 60-69 70-76 D
1 < 30% 0-59 ≤ 69 F

Table 15. Descriptive scales based on percentages

PERCENTAGE DESCRIPTION
90-100 Exceptional
80-89 Excellent
70-79 Good
60-69 Satisfactory
50-59 Barely acceptable
0-49 Unacceptable

The PtT values of any given completed research projects, considering all the various
research categories/subcategories, can be combined to form one integrated value. These
values can be used to evaluate the effectiveness of the agency’s entire research program.
The two basic approaches to combine these different PtT values generally are the
arithmetic mean and the weighted mean. The arithmetic mean approach considers all the
equal weights of the PtT values regardless of the RCs/RSCs for the completed research.
The weighted mean approach provides different weights for the PtT values based on the
relative importance of the RCs/RSCs in the agency’s research program. The

— 96 —
transportation agency can assign different weights at the level of RC, RSC, or both. To
demonstrate the processes for the weighted mean approach in three conditions such as
weights assigned to RCs, RSCs, or both, pseudo data for the PtT values and weights were
created. Table 16 is presented, if a state DOT has multiple complete research projects in
the different RSCs evaluated in Year 2018.

Table 16. Research evaluated for PtTs in year 2018

RC RSC Research Completed PtT (%)

Pavement/Material/Geotech
Design (D) D-1 1 95
(D-1)
D-1 2 58
Bridge/Structure (D-2) D-2 1 48
D-2 2 72
Road Design (D-3) D-3 1 88
D-3 2 69
Hydraulics (D-4) D-4 1 81
Project Management/
Construction
Construction Methods (CE- CE-1 1 63
Engineering (CE)
1)
CE-1 2 66
CE-1 3 82
CE-1 4 89
Product Development (CE-
None -
2)
Planning (P) Traffic/Roadway (P-1) P-1 1 52
P-1 2 42
Public Transportation (P-2) P-2 1 81
P-2 2 87
Administration/Workforce
None -
(P-3)
Maintenance/
Roadway (MO-1) MO-1 1 86
Operation (MO)
MO-1 2 58
MO-1 3 96
MO-1 4 70
MO-1 5 46
Transit (MO-2) MO-2 1 54
Safety and Security
Transport Safety (SS-1) SS-1 1 79
(SS)
SS-1 2 89
Transport Security (SS-2) None -

— 97 —
 RC RSC Research Completed PtT (%)
Sustainability (S) Environment (S-1) S-1 1 67
S-1 2 82
S-1 3 93
Society (S-2) S-2 1 79
S-2 2 55
Economy (S-3) None -

Table 17 shows the weights assigned to the RCs, which implies that the RSCs under each
RC have an equal weight. Therefore, the integrated PtT value was computed as 75.6% as
presented in Table 18. The mean PtT of the research for each RC in Table 18 is the
arithmetic mean of the PtT values of the complete research falling into the same RC. The
weighted PtT value for each RC in Table 18 is a product of the RC weight and the mean
PtT of research.

Table 17. Weights for the RCs

RC RC Weight (%) RSC

Design 15 Pavement Material/Geotech
Bridge/Structure
Road Design
Hydraulics
Project Management/
Construction Engineering 10
Construction Methods
Product Development
Planning 10 Traffic/Roadway
Public Transportation
Administration/Workforce
Maintenance/ Operation 15 Roadway
Transit
Safety and Security 30 Transport Safety
Transport Security
Sustainability 20 Environment
Society
Economy

— 98 —
 Table 18. Weighted PtT values for RCs

Weighted PtT Value

RC RC Weight (%) Mean PtT at RC (%)
(%)
Design 15 73.0 11.0
Construction
10 75.0 7.5
Engineering
Planning 10 65.5 6.6
Maintenance/Operation 15 68.3 10.3
Safety and Security 30 84.0 25.2
Sustainability 20 75.2 15.0
Total 100 75.6

Table 19 presents the different weights assigned to RSCs, assuming an equal weight for
the RCs. The sum of the RSC weights in each RC is equal to 1. In Table 16, some of the
RSCs, such as Production Development, Administration/Workforce, Transport Security,
and Economy, were assumed to have no research evaluated for PtT in 2018. The RSC
weights in

Table 19 therefore were adjusted for the RSCs with the evaluated research as shown in
Table 20. The weights can be adjusted by distributing the weight of a RSC with no
research evaluated for the year over the rest of the RSCs based on their initial portions in
the same RC. For example, the weights of Environment and Society in Sustainability are
equally 30% so that the weight of Economy is equally distributed to these two RSCs,
raising their weights to 50% (= 30%+(40%)⁄2). Accordingly, the weighted PtT values at
the RSC level of RSC were calculated by multiplying the mean PtT at the RSC by the
adjusted RSC weights. The sum of the weighted PtT values of the RSCs for each RC is
the weighted mean PtT value for the RC. This concludes that the overall PtT value for the
research program of the state DOT is 72.6%, which is an arithmetic mean of the weighted
mean PtT values of the RSCs as shown in Table 20.

Table 19. Weights for the RSCs

RC RSC RSC Weight (%)

Design Pavement/Material/Geotech 20
Bridge/Structure 30
Road Design 30
Hydraulics 20

— 99 —
 RC RSC RSC Weight (%)
Project Management/
Construction Engineering 60
Construction Methods
Product Development 40
Planning Traffic/Roadway 40
Public Transportation 40
Administration/Workforce 20
Maintenance/Operation Roadway 60
Transit 40
Safety and Security Transport Safety 50
Transport Security 50
Sustainability Environment 30
Society 30
Economy 40

Table 20. Weighed mean PtTs at the level of RSC

Weighted
Mean PtT of Adj. RSC Mean PtT of
RC RSC PtT Value
RSC (%) Weight (%) RC (%)
(%)
Design Pavement/Material/Geotech 76.5 20 15.3
Bridge/Structure 60.0 30 18.0
73.1
Road Design 78.5 30 23.6
Hydraulics 81.0 20 16.2
Construction Project Management/
75.0 100 75.0
Engineering Construction Methods 75.0
Production Development — — —
Planning Traffic/Roadway 47.0 50 23.5
Public Transportation 84.0 50 42.0 65.5
Administration/Workforce — — —
Maintenance/
Roadway 71.2 60 42.7
Operation 64.3
Transit 54.0 40 21.6
Safety and
Transport Safety 84.0 100 84.0
Security 84.0
Transport Security — — —
Sustainability Environment 80.6 50 40.3
73.8
Society 67.0 50 33.5
Economy — — —

— 100 —
For the last condition, the weights used in Table 18 and Table 20 were considered for the
RCs and RSCs as summarized in Table 21. The adjusted weights for the RSCs with the
research evaluated also were included in Table 21. As the PtT values of the complete
research in Table 16 were used, the weighted PtT values of the RSCs and the mean PtT
for RCs in Table 20 were utilized. The weighted PtT values for the RCs were calculated
as the product of the RC weights and the mean PtT values of the RCs as shown in Table
22. The sum of these weighted PtT values is the weighted mean PtT value for the
research program of the state DOT, which is 75.6%.

Table 21. Weights for both RCs and RSCs

RSC Weight Adj. RSC

RC RC Weight (%) RSC
(%) Weight (%)
Design 15 Pavement/Material/Geotech 20 20
Bridge/Structure 30 30
Road Design 30 30
Hydraulics 20 20
Construction Project Management/
10 60 100
Engineering Construction Methods
Product Development 40 —
Planning 10 Traffic/Roadway 40 50
Public Transportation 40 50
Administration/Workforce 20 —
Maintenance/
15 Roadway 60 60
Operation
Transit 40 40
Safety and
30 Transport Safety 50 100
Security
Transport Security 50 —
Sustainability 20 Environment 30 50
Society 30 50
Economy 40 —

— 101 —
 Table 22. Weighted PtT values for the RC

Weighted PtT Value

RC RC Weight (%) Mean PtT of RC (%)
(%)
Design 15 73.1 11.0
Construction
10 75.0 7.5
Engineering
Planning 10 65.5 6.6
Maintenance/Operation 15 64.3 9.6
Safety and Security 30 84.0 25.2
Sustainability 20 73.8 14.8
Total 100 74.7

— 102 —
 Conclusions and Recommendations

Public agencies at any level (e.g., federal, state, or local) spend large amounts of
resources on operating research programs to develop research that is essential to
improving transportation practices and policies that address current and future issues, as
well as finally achieve their intended goals. Therefore, successfully monitoring and
tracking the value of research is essential to promoting the process efficiency and quality
of a research program. However, estimating the value of complete research is challenging
due to the obstacles as follows: the difficulties in accurately describing intangible benefits
and interpreting qualitative benefit; the nature of wide-ranging research outputs, which
requires a variant form of techniques and measures for estimating the research values;
and the scarcity of data about the measures chosen to estimate research values. In an
attempt to mitigate the obstacles, this research was conducted to develop the mapping
table and research value estimation method through the data collection and analysis.

The data utilized for this research were derived from a comprehensive literature review
and two national surveys. The data were examined using thematic analysis and the
clustering/coding approach to generate the mapping table which includes the columns for
research categories, research subcategories, benefit categories, benefit subcategories,
benefit measures, and data types. The mapping table allows transportation agencies to
develop and recognize research and benefit categories/subcategories to identify
associated benefit measures and data types. Consequently, the data types are quantified
and integrated using the research value estimation method so that the value of complete
research can be efficiently measured and rational decisions about future program
development can be made.

The research deliverables (e.g., mapping table, guidebook, and final report) will enhance
the capability of state DOTs to operate and manage their research programs efficiently
and systematically. However, the use of the research findings can be further expanded by
the future efforts and recommendations as follows:

• Target values for benefit measures. The evaluation method equation

requires target values as an input to evaluate the performance of complete
research outcomes. Therefore, it is strongly recommended that state DOTs
conduct future work to develop systematic procedures to set up target values
for each of the benefit measures in the mapping table.

— 103 —
• Benefit measures not suitable to set up target values. Although the target
values are required to utilize the deliverables of this research best, it is not
always the case that a state DOT can set up target values for all benefit
measures because the associated data types may not be available and pursue
maximum or minimum (e.g., maximizing cost savings or minimizing life-
cycle costs). Therefore, the recommendations and future work to minimize the
possible negative impacts of these data types on the evaluation reliability are:
o To use a substitute research subcategory with a data type available for the
state DOT on a temporary basis, while the state DOT makes a strategic
plan to collect the originally-suggested data type as a permanent future
solution.
o To build historical data for the data types pursuing maximum or minimum
so that a statistical method, such as the Z-score approach (see the
guidebook for details), can be utilized.

• Mapping table as an evolving document. The mapping table was developed

considering all current practices of the state DOTs referred to this research. As
the current practices and policies could be revised or new practices and
policies could be added to the existing research programs, the mapping table
should be evaluated and updated on (if needed) a regular basis to
accommodate these possible changes.

• Research areas of completed projects not included in the mapping table.

Due to the diversity of the research areas within which research projects can
be developed, the mapping table is not able to incorporate every research area.
The mapping table has about a 95% coverage rate for all research areas being
used in state DOTs. For the 5% of research areas that are not included, it is
recommended that a state DOT first explore the objectives and expected
outcomes of the projects and match them into the benefit categories,
subcategories, and measures in the mapping table in order to the best-fit
research categories and subcategories.

— 104 —
 Acronyms, Abbreviations, and Symbols

Term Description
ASAP Alabama Service and Assistance Patrol
AADT Average Annual Daily Traffic
American Association of State Highway and Transportation Officials
AASHTO RAC
Research Advisory Committee
BCR Benefit Cost Ratio
CMF Crash Modification Factor
CO Carbon Monoxide
CO2 Carbon Dioxide
CR Coverage Rate
DOT Department of Transportation
EDC Every Day Counts
FHWA Federal Highway Administration
FRP Fiber-Reinforced Plastic
FY Fiscal Year
IE Implementation Effectiveness
ITS Intelligent Transportation System
LCA Life-Cycle Analysis
LOS Level of Service
NASBO National Association of State Budget Officers
NCHRP National Cooperative Highway Research Program
NDT Non-Destructive Test
NOx Oxides of Nitrogen
OMB Office of Management and Budget
PM Particulate Matter
PtT Progress towards Target
RPM Research Performance
SD Structurally Deficient
SHRP Strategic Highway Research Program

— 105 —
Term Description
SPFs Safety Performance Functions
STC Southeast Transportation Consortium
TAC Technical Advisory Committee
VOC Volatile Organic Compound

— 106 —
 References

1) Cambridge Systematics, Inc. (1999). Multimodal Transportation:

Development of a Performance Based Planning Process. National Cooperative
Highway Research Program 26, TRB, National Research Council,
Washington, D.C.
2) Cambridge Systematics, Inc. (2000). A Guidebook for Performance-Based
Transportation Planning. National Cooperative Highway Research Program
446, TRB, National Research Council, Washington, D.C.
3) Chase, C., PB Consult, Inc. and Texas Transportation Institute. (2006).
Performance Measures and Targets for Transportation Asset Management.
National Cooperative Highway Research Program 511, TRB, Washington,
D.C.
4) Chattopadhyay, R., and Ghosh, A. (2012). “Performance appraisal based on a
forced distribution system: its drawbacks and remedies.” International Journal
of Productivity and Performance Management, 61(8). 881-896.
5) Codd, N., and Walton, C.M. (1995). Performance Measures and a Framework
for Decision-Making under the National Transportation System.
SWUTC/95/465-1, Office of the Governor of the State of Texas, Energy
Office, Austin, Texas.
6) Czerniak, R., Gaiser, S., and Gerard, D. (1996). The Use of Intermodal
Performance Measures by State Departments of Transportation. U.S
Department of Transportation-T-96-18, Federal Highway Administration,
Washington, D.C.
7) Dai, F. and Yoon, Y. (2017). “Mapping Procedures for Value Determination of
Transportation Research Results.” American Society for Engineering
Management, 2017 International Annual Conference, October 18-21,
Huntsville, AL.
8) Gordon Proctor & Associates, Cambridge Systematics, American
Transportation Research Institute, StarIsis Corporation, and Council of Supply
Chain Management Professionals. (2011). Performance Measures for Freight
Transportation. National Cooperative Freight Research Program 10, TRB, The
National Academies Press, Washington, D.C.
9) Krugler, P., Walden, M. N., Hoover, B., Lin, Y. D., and Tucker, S. (2006).
Performance Measurement Tool Box and Reporting System for Research

— 107 —
 Programs and Projects. NCHRP 127 (web-only), National Cooperative
Highway Research Program, Washington, DC.
10) Litman, T. (2016). Well Measured: Developing Indicators for Sustainable and
Livable Transport Planning. Victoria Transport Policy Institute, Victoria, BC,
Canada.
11) Lomax, T.J., Zmud, J.P., Zmud, M., Schofer, J.L., Paasche, J.L., and Meyer.
J.A. (2009). Communicating the Value of Transportation Research. National
Cooperative Highway Research Program 610, TRB, The National Academies
Press, Washington, D.C.
12) Park, S.H., Han, S.H., and Chae, M.J. (2012). “Quantitative Performance
Measurement for Construction R&D Projects.” KSCE Journal of Civil
Engineering, 17(4), 610-619.
13) Ramani, T., Potter, J., DeFlorio, J., Reeder, V., and Zietsman, J. (2011). A
Guidebook for Sustainability Performance Measurement for Transportation
Agencies. National Cooperative Highway Research Program 708, TRB, The
National Academies Press, Washington, D.C..
14) Schiemer, M.J. (2015). “Transportation Operations Performance Measurement
& Management.” ITS Heartland – 2015 Annual Meeting, April 27 - 29,
Omaha, NE.
15) Shaw, T. (2003). Performance Measures of Operational Effectiveness for
Highway Segments and Systems. National Cooperative Highway Research
Program Synthesis 311, TRB, National Research Council, Washington, D.C.
16) Stewart, S.M., Gruys, M.L., and Storm, M. (2010). “Forced distribution
performance evaluation systems: Advantages, disadvantages and keys to
implementation.” Journal of Management & Organization, 16, 168–179.
17) United States Environmental Protection Agency (EPA). “Guide to Sustainable
Transportation Performance Measures.” EPA 231-K-10-004, August 2011.
18) Yoon, Y. and Dai, F. (2017) “Development of a Mapping Table for the Value
Determination of Transportation Research Results.” Journal of Management
in Engineering, 34(1), 04017043-1-12.

— 108 —
 Appendix
Note: Appendix A1–B2 are presented in a separate PDF document due to its size. To access
these appendixes, please contact Dr. Yoojung Yoon at [email protected] or
304.293.9937.

Appendix A1: Survey-1 Questionnaire

Appendix A2: Survey-1 Responses
Appendix B1: Survey-2 Questionnaire
Appendix B2: Survey-2 Responses

— 109 —
 Appendix C
Mapping Table (1)
RC RSC BC BSC Measure Data Type
Lifecycle Remaining Life Percent of design life achieved (1)
Pavement/ Improved
Reduction of Life-Cycle Cost Data of life-cycle cost
Material/ Infrastructure Economic
Monetary Savings Material cost per sq. yd, lane mile, pavement lifetime
Geotech
(D-1) Volume of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Remaining Life Percent of design life achieved (1)
Lifecycle Data of all current element value and all initial element
Average Health Index
Improved value (2)
Bridge/ Infrastructure Reduction of Life-Cycle Cost Data of life-cycle cost
Structure
Economic Data of historical installing time; cost of installation per
(D-2) Monetary Saving
min/hour/day
Volume of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Design Reduction in daily travel time; Reduction in travel time
(D) of goods to essential markets (region wide); Attract
Improved Accessibility to Essential
Capacity greater number of enterprises in key industries with
Accessibility/Mobility Destinations
enhanced accessibility to high-capacity highways or
rail facilities
Road Design Lifecycle Remaining Life Percent of design life achieved (1)
Improved
(D-3)
Infrastructure Economic Financial Measures Infrastructure maintenance cost (3)
Improved Road Networks Predictable Existence of a system that adjusts speeds based on the
Prevention Activity
Safety/Security and Recognizable to Users presence of alternative modes and context
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Water Resources/ The Amount of Water
Improved Environment Data of amount of water discharge
Hydraulics Flood Protection Discharged from the System
(D-4) Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Project Regionally Produced
Total weight/volume/cost of purchased materials
Construction Management/ Improved Construction Materials
Engineering Construction Project/Program Economic
Construction Costs within
(CE) Methods Delivery Total cost of materials and goods
Planned Budget
(CE-1)
 Mapping Table (2)

RC RSC BC BSC Measure Data Type

Impact of Construction
Travel time delay for commuters due to construction
Activities on System
Improved Accomplishment: activities
Efficiency
Project/Program Effectiveness,
Project Efficiency Duration of construction (4)
Delivery Efficiency, Quality
Percent of construction projects completed on
Delivery Rate
schedule/on budget
Change in number of crashes per time until within a
Crash Risk in Work Zone
Improved particular work zone
Prevention Activity
Project Safety/Security Improved Work Zone Traffic Change in the percentage use of traveler information
Management/ Control Activities systems
Construction Customer User Satisfaction towards
Methods Satisfaction/Quality of Construction Decisions that Percentage of customers satisfaction (5)
Construction (CE-1) Life Impact The Environment
Engineering Reduction of Activities Engine hours of operation; Emission rates; Idling hours
Air Quality/Emissions
(CE) Improved Environment generating Pollutant Emissions per day
Habitat Restoration and
Land Use Ratio of restored and maintained area to disturbed area
Landscaping
Number of noise receptor sites above established
Noise Impacts
standards (6)
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Improved Safety
Prevention Activity Accident Reduction Rate Number of accidents in a certain time period
/Security
Product
Improved Project/
Development Economic Monetary Savings Total cost of project
Program delivery
(CE-2)
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Change in number of jobs within reasonable travel time
Economic Accessibility to Jobs
(by mode) for region's population
Traffic/
Planning Improved Customer Satisfaction Customer perception of satisfaction with commute
Roadway Quality of Life
(P) Accessibility/Mobility /Quality of Life time; Lost time due to congestion
(P-1)
Per capita; Per measure of regional business volume;
Capacity Total Freeway Lane-Miles
Per VMT; Per square mile
 Mapping Table (3)

RC RSC BC BSC Measure Data Type

Reduction of Activities Change in trips; VMT; Percent of non-drivers; Tons of
Air Quality/Emissions
generating Pollutant Emissions emissions per day
Economic Cost Efficiency Average cost per trip; Average cost per ton-mile
Number of residences or percent of population exposed
Improved Environment Noise Users Affected to noise/air/light pollution exceeding established
standards
Customer Perception towards
Customer Satisfaction
Transportation Decisions that Percentage of customers satisfaction
/Quality of Life
Impact the Environment
Economic Cost Per Accident Average accident cost per trip
Number of Safety Aspects
Prevention Activity Considered Early in Project Percentage of project implementing predictive methods
Traffic/ Improved Planning
Roadway Safety/Security Emergency Response Incident Response Average response time for emergency services
(P-1)
Customer Satisfaction Customer Perception of Safety
Percentage customer satisfaction
Planning /Quality of Life While on Roadway Systems
(P) Schedule and Budget
Economic Percentage projects completed on time and on budget
Adherence
Customer Satisfaction Customer Perception towards
Percentage of customer satisfaction
Improved Project /Quality of Life Emergency Response Time
/Program Delivery Vehicle miles of detour due to projects; Hours of delay
Accomplishment: Customer Impact
due to projects
Effectiveness,
Percent of capital costs spent; Percent cost of
Efficiency, Quality Efficiency
preliminary engineering rework
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Percent of population with convenient access to public
Capacity Service Accessibility
transit
Public
Improved Economic Cost Efficiency Total transit operating expenditures per transit-mile
Transportation
Accessibility/Mobility Customer Ratings towards
(P-2) Customer Satisfaction
Transit Reliability, Percentage of customer satisfaction
/Quality of Life
Congestion, Cost, Time, etc.
 Mapping Table (4)

RC RSC BC BSC Measure Data Type

Change in planned miles/transit routes/pedestrian
Air Quality/Emissions Use of Non-motorized Modes facilities/population within reasonable distance of
transit
Demand for Single-Occupancy
Economic Change in the number or cost of multimodal options
Vehicle Travel
Improved Environment Noise Identify number of noise receptor sites above threshold
User Affected
(6)
Customer Perception towards
Customer Satisfaction Public Transportation
Percentage of customer satisfaction
/Quality of Life Decisions Which Impact the
Environment
Economic Cost of Incidents Average accidents cost per intermodal movement
Public
Transportation Prevention Activity Condition of Transit Systems Number of safety related improvements
(P-2) Improved Emergency Response Incident Response Response time to incidents
Safety/Security Customer Perception towards
Customer Satisfaction
Safety in Public Transportation Percentage of customers satisfied (5)
Planning /Quality of Life
Systems
(P)
Schedule and Budget
Economic Percentage projects completed on time and on budget
Adherence
Customer Satisfaction Customer Perception of
Improved Project Percentage of customer satisfaction (5)
/Quality of Life Emergency Response Time
/Program Delivery
Accomplishment:
Percent of capital costs spent; Percent cost of
Effectiveness, Efficiency
preliminary engineering rework
Efficiency, Quality
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Number of Entries Into Number of planned and actual entry level; Total
Industry Demand
Workforce number of employees over time
Improved
Education Supply Education Percentage passing each stage of screening
Administration/ Administration/Workfo
Economic Financial Measures Administrative costs as percent of total program
Workforce rce
(P-3) Improved Efficiency
Efficiency Total amount of time saved measured in hours
/Effectiveness
Volume Of Information
Increased knowledge Knowledge Number of citations, downloads, etc.
Sharing
 Mapping Table (5)

RC RSC BC BSC Measure Data Type

Improved Economic Financial Measures Infrastructure maintenance cost
Infrastructure System Preservation Preservation Activities Maintenance hours
Economic Financial Measures Public cost for transportation system
Customer Perception of
Improved Customer Satisfaction
Facility Operations and Percentage of customer satisfaction (5)
Accessibility/Mobility /Quality of Life
Availability
Capacity Availability Number of hours of road closure
Air Quality/Emissions Fuel Efficiency Average fuel consumption per VMT
Tons of pollution (or vehicle emissions) generated per
Economic Financial Measures
day
Roadway
Improved Environment Noise Impacts Number of noise receptor sites above threshold (6)
(MO-1)
Customer Perception of
Customer Satisfaction
Operational Impacts on Percentage of customer satisfaction (5)
/Quality of Life
Environment
Economic Financial Measures Number of fatalities; Reduced congestion
Maintenance/ Prevention Activity General Prevention Number of annual incidents
Operation Improved
Safety/Security Emergency Response Incident Recovery Incident clearance time
(MO)
Customer Satisfaction Customer Perception of
Percentage of customer satisfaction (5)
/Quality of Life Operational Safety
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Improved Economic Financial Measures Infrastructure maintenance cost
Infrastructure System Preservation Preservation Activities Maintenance hours
Economic Financial Measures Public cost for transportation system
Customer Perception of
Improved Customer Satisfaction
Facility Operations and Percentage of customer satisfaction (5)
Transit Accessibility/Mobility /Quality of Life
Availability
(MO-2)
Capacity Availability Number of hours of system/route closure
Reduction of Activities
Air Quality/Emissions Tons of emissions per day
generating Pollutant Emissions
Improved Environment
Economic Financial Measures Delays measured in time; Occurrence of incidents
Noise Impacts Number of noise receptor sites above threshold
 Mapping Table (6)

RC RSC BC BSC Measure Data Type

Customer Perception of
Improved Customer Satisfaction
Operational Impacts on Percentage of customer satisfaction (5)
Environment /Quality of Life
Environment
Economic Financial Measures Delays measured in time; Occurrence of incidents
Percent of facilities with specific security features
Maintenance/ Prevention Activity Security Improvements
Transit (cameras, lighting, guards, etc.)
Operation Improved
(MO-2) Emergency Response Incident Recovery Incident clearance time
(MO) Safety/Security
Customer Satisfaction Customer Perception of
Percentage of customer satisfaction (5)
/Quality of Life Operational Safety
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Customer Perception of
Safety While on Roadway percentage customer satisfaction (5)
Customer Satisfaction Systems
/Quality of Life Customer Perception of
Safety While on Public percentage customer satisfaction
Transport Improved Safety Transportation Systems
Safety
Economic Financial Measures Delays measured in time; Occurrence of incidents
(SS-1)
Prevention Activity General Prevention Number of roadway sections; Number of roadway
Funds Availability towards
Emergency Response Response time to incidents
Safety and Incidents
Security Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
(SS) Sharing
Capacity of Transportation
Customer Satisfaction Change in capacity of parallel/redundant routes
Systems to Recover Swiftly
/Quality of Life across all modes
from Incidents
Transport Improved Security Economic Financial Measures Delays measured in time; Occurrence of incidents
Security Percent of facilities with specific security features
Prevention Activity Security Improvements
(SS-2) (cameras, lighting, guards, etc.)
Emergency Response Incident Recovery Incident clearance time
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
 Mapping Table (7)

RC RSC BC BSC Measure Data Type

Customer Perception of
Satisfaction with
Transportation Decisions
Customer Satisfaction
Which Impact on Percentage of customers satisfaction
/Quality of Life
Environment/Air Quality/
Water Quality/Noise
Conditions
Reduction of Activities
Air Quality/Emissions Tons of emissions per day
Improved Environment generating Pollutant Emissions
Environment Water Pollution; Ability to
(S-1) Water Resources /Flood Per capita fuel consumption; Per capita impervious
Minimize Impervious Surface
Protection surface area
Area
Open Space and Biodiversity
Land Use Per capita land devoted to transport facilities
Protection
Noise Impacts Number of noise receptor sites above threshold (6)
Sustainability Per capita transport energy consumption; Per capita use
(S) Economic Energy Costs
of imported fuels
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Transportation Systems to Transport system diversity; Portion of destinations
Improved Customer Satisfaction Accommodate All Users accessible by people with disabilities and low incomes
Accessibility/Mobility /Quality of Life Community Development Quality of road and street environments
Society Cultural Heritage Preservation Responsiveness to traditional communities
(S-2) Improved Customer Satisfaction Per capita traffic casualty (injury and death) rates;
Risk of Accidents
Safety/Security Quality of Life Traveler assault (crime) rates
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
Customer Satisfaction Essential Services and Portion of low-income households that spend more than
Economy Improved /Quality of Life Activities 20% of budgets on transport
(S-3) Accessibility/Mobility Pricing and Incentives Per capita GDP/ Per capita congestion delay; Efficient
Economic
/Maximize Accessibility prioritization of facilities
 Mapping Table (8)

RC RSC BC BSC Measure Data Type

Operations and Asset
Improved Project
Economic Management Maximizes Cost Service delivery unit costs compared with peers
Sustainability Economy /Program Delivery
Efficiency
(S) (S-3)
Volume Of Information
Increased Knowledge Knowledge Number of citations, downloads, etc.
Sharing
 Appendix D
Gap Analysis Results (1)
RC RSC BC Measure LA VA GA FL MS WV NC
Remaining Life Y Y Y Y Y
Pavement/Material Improved Infrastructure Reduction of Life-Cycle Cost Y Y Y
/Geotech Monetary Savings Y Y Y
Increased Knowledge Volume of Information Sharing
Remaining Life Y Y
Average Health Index Y
Improved Infrastructure
Bridge/Structure Reduction of Life-Cycle Cost
Monetary Saving
Design Increased Knowledge Volume of Information Sharing
Improved
Accessibility to Essential Destinations Y Y Y Y Y Y
Accessibility/Mobility
Remaining Life Y Y Y Y Y
Road Design Improved Infrastructure
Financial Measures Y Y Y Y Y
Improved Safety/Security Road Networks Predictable and Recognizable to Users Y Y Y Y Y
Increased Knowledge Volume Of Information Sharing
Improved Environment The Amount of Water Discharged from the System Y Y Y Y
Hydraulics
Increased Knowledge Volume Of Information Sharing
Regionally Produced Construction Materials Y
Construction Costs within Planned Budget Y
Improved Project/Program
Impact of Construction Activities on System Efficiency Y Y Y
Delivery
Project Efficiency Y Y
Delivery Rate Y Y
Crash Risk in Work Zone Y Y Y Y Y Y
Project Management/ Improved Safety/Security
Construction Construction Methods Improved Work Zone Traffic Control Activities Y Y Y Y Y Y
Engineering User Satisfaction towards Construction Decisions Which
Y Y Y Y
Impact the Environment
Improved Environment Reduction of Activities generating Pollutant Emissions Y Y Y Y
Habitat Restoration and Landscaping Y Y Y Y
Impacts Y Y Y Y
Increased Knowledge Volume Of Information Sharing
Product Development Improved Safety/Security Accident Reduction Rate Y Y Y Y Y Y
 Gap Analysis Results (2)

RC RSC BC Measure LA VA GA FL MS WV NC
Improved Project/ Program
Construction Monetary Savings Y
Product Development delivery
Engineering
Increased Knowledge Volume Of Information Sharing
Accessibility to Jobs Y
Improved
Quality of Life Y
Accessibility/Mobility
Total Freeway Lane-Miles Y Y
Reduction of Activities generating Pollutant Emissions Y Y Y Y
Cost Efficiency Y Y Y Y
Improved Environment Users Affected Y Y Y Y
Customer Perception towards Transportation Decisions
Y Y Y Y
Which Impact the Environment
Cost Per Accident Y Y Y Y Y Y Y
Traffic/Roadway Number of Safety Aspects Considered Early in Project
Y Y Y Y Y Y Y
Planning
Improved Safety/Security
Incident Response Y Y Y Y Y Y
Customer Perception of Safety While on Roadway
Y Y Y Y Y Y
Systems
Planning Schedule and Budget Adherence
Improved Project/Program Customer Perception towards Emergency Response Time
Delivery Customer Impact Y Y
Efficiency Y
Increased Knowledge Volume Of Information Sharing
Service Accessibility Y Y
Improved Cost Efficiency Y
Accessibility/Mobility Customer Ratings towards Transit Reliability,
Y
Congestion, Cost, Time, etc.
Public Transportation Use of Non-motorized Modes Y Y Y Y
Demand for Single-Occupancy Vehicle Travel Y Y Y Y
Improved Environment User Affected Y Y Y Y
Customer Perception towards Public Transportation
Y Y Y Y
Decisions Which Impact the Environment
 Gap Analysis Results (3)

RC RSC BC Measure LA VA GA FL MS WV NC
Cost of Incidents Y Y Y Y Y Y Y
Condition of Transit Systems Y Y Y Y Y Y Y
Improved Safety/Security Incident Response Y Y Y Y Y Y
Customer Perception towards Safety in Public
Y Y Y Y Y Y
Public Transportation Transportation Systems
Schedule and Budget Adherence
Improved Project/Program
Customer Perception of Emergency Response Time
Planning Delivery
Efficiency Y
Increased Knowledge Volume Of Information Sharing
Number of Entries Into Workforce Y
Improved Education Y
Administration/
Administration/Workforce Financial Measures Y Y
Workforce
Efficiency
Increased knowledge Volume Of Information Sharing
Financial Measures Y Y Y Y
Improved Infrastructure
Preservation Activities Y Y Y Y
Financial Measures Y
Improved Customer Perception of Facility Operations and
Y
Accessibility/Mobility Availability
Availability Y Y
Fuel Efficiency Y Y Y Y
Financial Measures Y Y Y Y
Roadway
Maintenance/ Improved Environment Impacts Y Y Y Y
Operation Customer Perception of Operational Impacts on
Y Y Y Y
Environment
Financial Measures Y Y Y Y Y Y Y
General Prevention Y Y Y Y Y Y Y
Improved Safety/Security
Incident Recovery Y Y Y Y Y Y
Customer Perception of Operational Safety Y Y Y Y Y Y
Increased Knowledge Volume Of Information Sharing
Financial Measures Y Y Y Y
Transit Improved Infrastructure
Preservation Activities Y Y Y Y
 Gap Analysis Results (4)

RC RSC BC Measure LA VA GA FL MS WV NC
Financial Measures Y
Improved Customer Perception of Facility Operations and
Y
Accessibility/Mobility Availability
Availability Y Y
Reduction of Activities generating Pollutant Emissions Y Y Y Y
Financial Measures Y Y Y Y Y
Maintenance/ Improved Environment Impacts Y Y Y Y
Transit
Operation Customer Perception of Operational Impacts on
Y Y Y Y
Environment
Financial Measures Y Y Y Y Y Y Y
Security Improvements Y Y Y Y Y Y
Improved Safety/Security
Incident Recovery Y Y Y Y Y Y
Customer Perception of Operational Safety Y Y Y Y Y Y
Increased Knowledge Volume Of Information Sharing
Customer Perception of Safety While on Roadway
Y Y Y Y Y Y
Systems
Customer Perception of Safety While on Public
Y Y Y Y Y Y
Improved Safety Transportation Systems
Transport Safety
Financial Measures Y Y Y Y Y Y Y
General Prevention Y Y Y Y Y Y
Safety and Funds Availability towards Incidents Y Y Y Y Y Y Y
Security Increased Knowledge Volume Of Information Sharing
Capacity of Transportation Systems to Recover Swiftly
from Incidents
Improved Security Financial Measures Y
Transport Security
Security Improvements
Incident Recovery
Increased Knowledge Volume Of Information Sharing
Customer Perception of Satisfaction with Transportation
Sustainability Environment Improved Environment Decisions Which Impact the Environment/Air Y Y Y Y
Quality/Water Quality/Noise Conditions
 Gap Analysis Results (5)

RC RSC BC Measure LA VA GA FL MS WV NC
Reduction of Activities generating Pollutant Emissions Y Y Y Y
Water Pollution; Ability to Minimize Impervious
Y Y Y Y
Surface Area
Improved Environment
Environment Open Space and Biodiversity Protection Y Y Y Y
Impacts Y Y Y Y
Energy Costs Y Y Y Y
Increased Knowledge Volume Of Information Sharing
Transportation Systems to Accommodate All Users Y
Improved
Sustainability Community Development Y
Accessibility/Mobility
Society Cultural Heritage Preservation Y
Improved Safety/Security Risk of Accidents Y Y Y Y Y Y Y
Increased Knowledge Volume Of Information Sharing
Improved Essential Services and Activities Y Y
Accessibility/Mobility Pricing and Incentives/Maximize Accessibility Y
Economy Improved Project/Program Operations and Asset Management Maximizes Cost
Y
Delivery Efficiency
Increased Knowledge Volume Of Information Sharing
 Appendix E
General Definitions of the Research Categories and Subcategories (1)
Research Category Research Subcategory General Definition
- Pavement design for new roadways and the rehabilitation of existing roadways to optimize the level of service
provided to road users.
- Material design to meet the structural and functional demands of roadways.
Pavement/Material/
- Geotechnical issues in pavement and material design.
Geotech
- For example, pavement structure design; material testing; specification; mixture design and optimization for
both traditional and innovative materials; characterization of subgrades and unbound base and subbase
materials.
Design - Design for bridges and all other transportation structures such as railroads, parking facilities, docks, and bus
Bridge/Structure
stations.
- Geometric design of roads concerned with the positioning of the road physical elements to optimize the
Road functionality of roads for efficiency and safety.
- For example, designs for alignment, profile, and cross section.
- Hydraulic design to ensure that transportation structures
Hydraulics (e.g., roadways, bridges, and railroads) have sufficient capabilities to control the movement of water and
handle water-related impacts (e.g., erosion, collapse, and sediment).
- Enhancement of the skills and knowledge of state DOT practitioners regarding project management and
construction methods to successfully deliver complete transportation projects on-time and within-budget
Project Management/ without sacrificing safety and quality.
Construction Construction Methods - For example, project data management; resource management; project delivery; partnering; effective
Engineering governance and culture innovation; managing and reporting on the delivery status; advancement in existing
construction technologies and materials; project cost, time, safety, and quality controls.
- Development of new technologies, materials, systems, and tools to enhance the efficiency and performance of
Product Development
transportation project planning, design, and construction.
- Planning to enhance the safe and efficient movement of pedestrians, cyclists, vehicles, and goods on roadways
Traffic/Roadway - For example, traffic impact analysis; traffic data collection; traffic concurrency studies; trip generation studies;
transportation due diligence; traffic operational analysis; corridor studies; traffic signal warrant studies.
Planning Public Transportation - Development of public transportation systems such as buses, trains, subways, and other forms of
Planning transportation that charge set fares, run on fixed routes, and are available to the public.
- Planning to optimize the quality of trips for public transportation users.
- For example, strategic policy development; demand forecasting; operational and business-case analysis;
network connectivity; route selection; urban-land-use issues.
 General Definitions of the Research Categories and Subcategories (2)
Research
Research Subcategory General Definition
Category
- Administration planning to enhance the operations and services of an administrative infrastructure
Administration/
Planning - Workforce planning to align the size, type, experience, knowledge, and skills of the agency’s workforce
Workforce
with the needs and priorities to achieve its goal.
- Maintenance and operations of physical roadway systems to maintain a quality of serviceability.
Roadway - This subcategory includes all physical facilities (e.g., bridges, tunnels, rest areas, and public areas) on the
Maintenance/ route of roadways.
Operation
- Maintenance and operations of various public transit systems which include railways and mass transit
Transit
systems.
Safety and Transport Safety - Protection of users from unintended structural and functional failures or errors of transportation systems.
Security Transport Security - Protection of users from deliberate or malicious attempts to disrupt or destroy transportation systems.
- Prolonged examination and preservation of the environment regarding factors such as water and air
Environment
quality.
Sustainability - Efforts to assure public equality, freedom and a healthy standard of living by developing transportation
Society
systems sustainably.
Economy - Development and enhancement of transportation systems in a way to reduce costs in economic sectors.
 Appendix F
Different Applications of the Data Types Numbered (1)
Data Types Numbered Applications
Percent of design life achieved (1) - Percent asset quantity with fewer than 5 years remaining service life (RSL)
- Average Remaining Service Life
- Percent of design life achieved
- Percent asset quantity forecast to achieve full design life
- Average age or percent asset quantity greater than “n” years old (age can be a useful proxy for remaining life when data are limited)
- Percent pavement miles with weight restrictions due to structural limitations
- Percent assets eligible for replacement
- Percent asset quantity out of service due to deteriorated condition
Data of all current element value and all - Average health index (0–100 scale)
initial element value (2) - Percent structurally deficient (SD)
- Percent with sufficiency rating less than 50
- Percent of bridges that meet department standards
- Number of posted or restricted bridges
- Number of steel bridges with section loss in a member
- Percent of bridges with deck, superstructure, and substructure NBI rating of 4 or below
Infrastructure maintenance cost (3) - Cost/benefit of existing facility vs. new construction
- Number and dollar value of projects that improve travel time on key routes
- Average cost per lane-mile constructed
- Cost per percentage point increase in lane miles rates fair or better on pavement condition
- Percentage of increase in final amount paid for completed construction over original contract amount
- Percent cost of re-work
- Construction Productivity index (Cost of contract lettings, utilities, real estate acquisition, construction, change orders, and cost
overruns DIVIDED BY staff costs, consultant contracts, and design construction change orders)
Duration of construction (4) - Cost per lane-mile constructed
- Administrative costs as percent of total program
- Preliminary engineering (PE) and construction engineering (CE) costs as percent of construction costs
- Design costs as percent of construction dollars let
- Percent of highway capital costs spent on construction (contractor payments and direct on-site construction oversight)
- Percent of cost of preliminary engineering rework
- Duration of construction (by project type)
Percentage of customer satisfaction (5) - Customer satisfaction with transportation decisions affecting the environment
- Customer perception of air quality
- Customer satisfaction rating for different maintenance elements
- Customer rating of asset condition or agency preservation activities
- Customer satisfaction rating
- Customer ratings of trip time, reliability, congestion severity, travel cost, travel time, etc.
- Customer satisfaction with snow and ice removal
 Different Applications of the Data Types Numbered (2)
Data Types Numbers Applications
Number of noise receptor sites above - Number of residences or percent of population exposed to highway noise exceeding established standards (or greater than
established standards (6) X decibels)
- Number of noise receptor sites above threshold • Constraints on use due to noise (or water)
- Percent of road network (including concrete sections) with quieter road surface by 2010
 Appendix G
Definitions for Measures in the Mapping Table (1)
RC RSC BC BSC Measure Definition
Condition and remaining life measures can be expressed as
Lifecycle Remaining Life averages or distributions (e.g., percent of system length or VMT
on roads in good, fair, and poor condition).
Sum of all recurring and non-recurring costs over the full life
Improved
span or a specified period of a good, service, structure, or
Infrastructure Reduction of Life-Cycle
D-1 system. Includes purchase price, installation cost, operating
Economic Cost
costs, maintenance and upgrade costs, and remaining (residual or
salvage) value at the end of ownership or its useful life.
Monetary Savings Monetary savings due to use of durable materials
Volume of Information
Increased Knowledge Knowledge
Sharing
Condition and remaining life measures can be expressed as
Remaining Life averages or distributions (e.g., percent of system length or VMT
on roads in good, fair, and poor condition).
Lifecycle
Condition and remaining life measures can be expressed as
Average Health Index averages or distributions (e.g., percent of system length or VMT
Improved on roads in good, fair, and poor condition).
D
Infrastructure Sum of all recurring and non-recurring costs over the full life
D-2
span or a specified period of a good, service, structure, or
Reduction of Life-Cycle
system. Includes purchase price, installation cost, operating
Economic Cost
costs, maintenance and upgrade costs, and remaining (residual or
salvage) value at the end of ownership or its useful life.
Monetary Saving Monetary saving due to reduction of installing time
Volume of Information
Increased Knowledge Knowledge
Sharing
Improved Accessibility to Essential Change in travel time (by mode) to schools, health services,
Capacity
Accessibility/Mobility Destinations grocery stores, civic and public spaces, recreation
Condition and remaining life measures can be expressed as
Lifecycle Remaining Life averages or distributions (e.g., percent of system length or VMT
D-3 on roads in good, fair, and poor condition).
Improved
Average cost per mile/ Average cost per trip /Vehicle operating
Infrastructure
cost reductions /Additional costs per trip (user fees) /Reduced
Economic Financial Measures
costs per trip (subsidies)/ Use cost/ Person-mile (user cost)/
Insurance costs/ Value of fuel savings
 Definitions for Measures in the Mapping Table (2)

RC RSC BC BSC Measure Definition

Improved Road Networks Predictable Existence of a functional class system of roadway; Existence of a
Prevention Activity
Safety/Security and Recognizable to Users system that adjusts speeds based on the hierarchy of roads.
D-3
Volume Of Information
Increased Knowledge Knowledge
Sharing
D
Water Resources/Flood The Amount of Water Volumetric flow rate of water that is transported through a given
Improved Environment
Protection Discharged from the System cross-sectional area.
D-4
Volume Of Information
Increased Knowledge Knowledge
Sharing
Regionally Produced Goods and materials should be purchased within a certain radius
Construction Materials from the project
Economic
Construction Costs within Proportion of projects with construction costs within planned
Planned Budget budget
Impact of Construction
Improved Activities on System Change in multimodal LOS due to construction activities
Project/Program Efficiency
Delivery Accomplishment:
Design costs as percent of construction dollars/ Administrative
Effectiveness, Project Efficiency
costs as percent of total program/ Cost per lane-mile constructed
Efficiency, Quality
Right of way plans completed on time/ Construction projects
Delivery Rate completed on schedule/ Construction projects completed on
CE CE-1 budget
Number of crashes as a portion of total time of work zones by
Crash Risk in Work Zone
Improved functional class, county, and district/region
Prevention Activity
Safety/Security Improved Work Zone Traffic Number of traffic control supervisors that are trained and on- site
Control Activities number of annual traveler safety complaints
User Satisfaction towards
Customer
Construction Decisions
Satisfaction/Quality of Refer to Appendix F
Which Impact the
Life
Improved Environment Environment
Reduction of Activities
Air Quality/Emissions generating Pollutant Tons of emissions generated per day due to construction
Emissions
 Definitions for Measures in the Mapping Table (3)

RC RSC BC BSC Measure Definition

Habitat Restoration and Acres of wetlands replaced or protected for every acre affected by
Land Use
Improved Environment Landscaping construction
CE-1 Noise Impacts Refer to Appendix F
Volume Of Information
Increased Knowledge Knowledge
Sharing
CE Improved
Prevention Activity Accident Reduction Rate Rate of accidents
Safety/Security
Improved Project/
CE-2 Economic Monetary Savings Equipment costs/labor costs/equipment lifecycle
Program delivery
Volume Of Information
Increased Knowledge Knowledge
Sharing
Economic Accessibility to Jobs Direct jobs created
Customer Percent of population that perceives that its environment has
Improved Satisfaction/Quality of Quality of Life become more 'livable' over the past year with regard to ability to
Accessibility/Mobility Life access desired activities
Capacity Total Freeway Lane-Miles Total freeway lane-miles in acceptable condition
Reduction of Activities
Change in percentage of commercial vehicles by EPA tier
Air Quality/Emissions generating Pollutant
compliance
Emissions
Ratio of oversize or overweight permit fees collected to dollar
Economic Cost Efficiency
value of damage caused
Improved Environment Noise Users Affected Refer to Appendix F
P P-1 Customer Perception towards
Customer
Transportation Decisions
Satisfaction/Quality of Refer to Appendix F
Which Impact the
Life
Environment
Economic Cost Per Accident Accident rate, deaths, injury, property loss by type of corridor
Number of Safety Aspects
Projects where safety of a project was reviewed in each of the
Prevention Activity Considered Early in Project
project development stages by a multidisciplinary review team
Improved Planning
Safety/Security Emergency Response Incident Response Response time to incidents
Customer Customer Perception of
Satisfaction/Quality of Safety While on Roadway Refer to Appendix F
Life Systems
 Definitions for Measures in the Mapping Table (4)

RC RSC BC BSC Measure Definition

Schedule and Budget
Economic Contracts completed on time
Adherence
Customer
Improved Customer Perception towards
Satisfaction/Quality of Refer to Appendix F
Project/Program Emergency Response Time
Life
P-1 Delivery
Accomplishment: Customer Impact Lane-hours restricted due to construction
Effectiveness,
Efficiency, Quality Efficiency Duration of project
Volume Of Information
Increased Knowledge Knowledge
Sharing
Capacity Service Accessibility Access time to intermodal facilities
Economic Cost Efficiency Cost per passenger
Improved
Accessibility/Mobility Customer Customer Ratings towards
Satisfaction/Quality of Transit Reliability, Refer to Appendix F
Life Congestion, Cost, Time, etc.
Route or service miles of transit routes, pedestrian facilities,
P designated bike facilities, population within 1 mile of transit,
person-miles walk distance to transit stops, person-miles distance
Air Quality/Emissions Use of Non-motorized Modes
from building entrances to public pedestrian facilities (sidewalks,
pedestrian ways), connectivity index (pedestrian facilities, bike
facilities, transit) due to project
P-2 Demand for Single-
Improved Environment Economic Make multimodal options cheaper and more accessible
Occupancy Vehicle Travel
Noise User Affected Refer to Appendix F
Customer Perception towards
Customer
Public Transportation
Satisfaction/Quality of Refer to Appendix F
Decisions Which Impact the
Life
Environment
Economic Cost of Incidents Accident rate, deaths, injury, property loss by type of corridor
Improved Miles of track not useable by certain traffic because of design or
Prevention Activity Condition of Transit Systems
Safety/Security condition deficiencies
Emergency Response Incident Response Response time to incidents
 Definitions for Measures in the Mapping Table (5)

RC RSC BC BSC Measure Definition

Customer Customer Perception towards
Improved
Satisfaction/Quality of Safety in Public Refer to Appendix F
Safety/Security
Life Transportation Systems
Schedule and Budget
Economic Contracts completed on time
Adherence
Customer
Improved Customer Perception of
P-2 Satisfaction/Quality of Refer to Appendix F
Project/Program Emergency Response Time
Life
Delivery
Accomplishment:
Effectiveness, Efficiency Duration of project
Efficiency, Quality
P Volume Of Information
Increased Knowledge Knowledge
Sharing
Number of Entries Into
Industry Demand Experienced hires
Workforce
Improved Education Supply Education Number of students enrolled in related fields of study
Administration/Workfor Preliminary engineering (PE) and construction engineering (CE)
P-3 ce Economic Financial Measures
costs as percent of construction costs
Improved Efficiency/
Efficiency Effort in administrative work
Effectiveness
Volume Of Information
Increased knowledge Knowledge
Sharing
Average cost per mile/ Average cost per trip /Vehicle operating
cost reductions /Additional costs per trip (user fees) /Reduced
Economic Financial Measures
costs per trip (subsidies)/ Use cost/ Person-mile (user cost)/
Improved Infrastructure
Insurance costs/ Value of fuel savings
Percentage of highway mainline pavement (or bridges) rated good
System Preservation Preservation Activities
or better
MO MO-1 Total public expenditures on modal systems (freight vs.
Economic Financial Measures
passenger)
Customer Customer Perception of
Improved
Satisfaction/Quality of Facility Operations and Refer to Appendix F
Accessibility/Mobility
Life Availability
Number of truck units, railroad cars, or containers that can be
Capacity Availability
stored at intermodal facility
 Definitions for Measures in the Mapping Table (6)

RC RSC BC BSC Measure Definition

Air Quality/Emissions Fuel Efficiency Gallons of wasted fuel
Economic Financial Measures Economic Costs of pollution
Noise Impacts Refer to Appendix F
Improved Environment
Customer Customer Perception of
Satisfaction/Quality of Operational Impacts on Refer to Appendix F
Life Environment
Economic Financial Measures Economic costs of incidents
Grade crossing collisions/ Level of redundancy for critical
Prevention Activity General Prevention passenger and freight infrastructure/ Change in the capacity of
MO-1
parallel routes across all modes
Improved Change in the number/ Value of projects as part of program
Safety/Security Emergency Response Incident Recovery designed to improve capacity of the transportation system to
recover swiftly from incidents
Customer
Customer Perception of
Satisfaction/Quality of Refer to Appendix F
Operational Safety
Life
Volume Of Information
Increased Knowledge Knowledge
MO Sharing
Average cost per mile/ Average cost per trip /Vehicle operating
cost reductions /Additional costs per trip (user fees) /Reduced
Economic Financial Measures
costs per trip (subsidies)/ Use cost/ Person-mile (user cost)/
Improved Infrastructure
Insurance costs/ Value of fuel savings
Percentage of highway mainline pavement (or bridges) rated good
System Preservation Preservation Activities
or better
Total public expenditures on modal systems (freight vs.
Economic Financial Measures
passenger)
MO-2 Customer Customer Perception of
Improved
Satisfaction/Quality of Facility Operations and Refer to Appendix F
Accessibility/Mobility
Life Availability
Number of railroad cars, or busses that can be stored at
Capacity Availability
intermodal facility
Reduction of Activities
Air Quality/Emissions Change in trips; VMT; percent non-driver
generating Pollutants
Improved Environment
Economic Financial Measures Economic costs of congestion
Noise Impacts Refer to Appendix F
 Definitions for Measures in the Mapping Table (7)

RC RSC BC BSC Measure Definition

Customer Customer Perception of
Improved Environment Satisfaction/Quality of Operational Impacts on Refer to Appendix F
Life Environment
Economic Financial Measures Economic costs of transit incidents
Prevention Activity Security Improvements Percent of facilities passing security tests
Change in the number/ Value of projects as part of program
MO MO-2 Improved
Emergency Response Incident Recovery designed to improve capacity of the transportation system to
Safety/Security
recover swiftly from incidents
Customer Satisfaction/ Customer Perception of
Refer to Appendix F
Quality of Life Operational Safety
Volume Of Information
Increased Knowledge Knowledge
Sharing
Customer Perception of
Safety While on Roadway Refer to Appendix F
Customer
Systems
Satisfaction/Quality of
Customer Perception of
Life
Safety While on Public Refer to Appendix F
SS SS-1 Improved Safety Transportation Systems
Economic Financial Measures Economic costs of transit incidents
Transit system stations not meeting safety standards/ Transit
Prevention Activity General Prevention
locations with identified hazards
Funds Availability towards Relative change in capital and operational funding allocated to
Emergency Response
Incidents disaster/ Incident response and management
Volume Of Information
SS-1 Increased Knowledge Knowledge
Sharing
Capacity of Transportation Change in the number/value of projects as part of program
Customer Satisfaction/
Systems to Recover Swiftly designed to improve capacity of the transportation system to
Quality of Life
from Incidents recover swiftly from incidents
Economic Financial Measures Economic costs of transit incidents
SS Improved Security
Prevention Activity Security Improvements Percent of facilities passing security tests
SS-2
Change in the number/value of projects as part of program
Emergency Response Incident Recovery designed to improve capacity of the transportation system to
recover swiftly from incidents
Volume Of Information
Increased Knowledge Knowledge
Sharing
 Definitions for Measures in the Mapping Table (8)

RC RSC BC BSC Measure Definition

Customer Perception of
Satisfaction with
Customer Satisfaction/
Transportation/ Air Quality/ Refer to Appendix F
Quality of Life
Water Quality/ Noise
Conditions
Reduction of Activities
Air Quality/Emissions Change in trips; VMT; Percent non-driver
generating Pollutants
Improved Environment
Water Pollution; Ability to
S-1 Water Resources/ Flood
Minimize Impervious Surface Management of used oil, leaks and stormwater
Protection
Area
Open Space and Biodiversity Support for smart growth development/Policies to protect high
Land Use
Protection value farmlands and habitat
Noise Impacts Refer to Appendix F
Economic Energy Costs Particularly petroleum imports
Volume of Information
Increased Knowledge Knowledge
Sharing
Transportation Systems to
S Those with disabilities, low incomes, and other constraints.
Customer Accommodate All Users
Improved
Satisfaction/Quality of
Accessibility/Mobility Community Development Walkability and bikability
Life
Cultural Heritage
S-2 Preservation of cultural resources and traditions
Preservation
Improved Customer Satisfaction/ Human exposure to harmful pollutants/Portion of travel by
Risk of Accidents
Safety/Security Quality of Life walking and cycling
Volume Of Information
Increased Knowledge Knowledge
Sharing
Customer Satisfaction/ Essential Services and Availability and quality of affordable modes (walking, cycling,
Improved Quality of Life Activities ridesharing and public transport)
Accessibility/Mobility Pricing and Incentives/ Portion of budgets devoted to transport/ Efficient pricing (road,
Economic
Maximize Accessibility parking, insurance, fuel, etc.)
S-3 Improved Operations and Asset
Project/Program Economic Management Maximizes Cost Performance audit results/ Service quality
Delivery Efficiency
Volume Of Information
Increased Knowledge Knowledge
Sharing
This public document is published at a total cost of $200.
29 copies of this public document were published in this first
printing at a cost of $200. The total cost of all printings of
this document including reprints is $200. This document was
published by Louisiana Transportation Research Center to
report and publish research findings as required in R.S. 48:105.
This material was duplicated in accordance with standards for
printing by state agencies established pursuant to R.S. 43:31.
Printing of this material was purchased in accordance with the
provisions of Title 43 of the Louisiana Revised Statutes.