Monitoring R&D functions

Techniques for Monitoring
R & D functions
Submitted By: Nandita Das
Dairy Chemistry (Ph.D)
Class Seminar On

CONTENTS
Introduction
Importance of monitoring R&D activities
What is monitoring?
Techniques of monitoring the R&D activities
Peer review/Expert management
Bibliometric methods- Counts and citation analysis
Bibliometric methods- Data mining
Bibliometric methods- Hotspot analysis
Survey method
Technology commercialization tracking model
Network analysis
Case study method
More tools
Conclusion

INTRODUCTION
WhatisR&D?
“R&D is the purposeful and systematic use of scientific
knowledge to improve man’s lot even though some of its
manifestations do not meet with universal approval .”
(Twiss, 1992)
“To develop new knowledge and apply scientific or engineering
knowledge to connect the knowledge in one field to that in
others.”
(Roussel et al., 1991)
R&D at different level
of business
R&D for existing
businesses.
This will ensure the businesses are able to compete and
exploit all opportunities available to it.
Drive new businesses.
Business opportunities will continually arise. R&D will ensure
that these can be exploited
Exploratory research
This helps to develop understanding of technology that the
business is using or may use

Defend, Support.
And Expand
Existing Business
Drive New
Business
Broaden/Deepen
Technological
Capability
R&D has three main strategic purposes to a business
The effective use of R&D naturally parallels the business cycle
for a product or industry
There are numerous Businesses R&D is into:
• Information and communication systems
• Electronic devices
• Power & Industrial systems
• Digital media & consumer products
• Logistic services
• Higher education institutes
• Medicine
• Food industries etc.

Goal of technology development R&D manager
 To complete research objectives that lead to successfully commercialized technologies.
 To continuously improve the program
 To communicate effectively to others for benefitting the program
What is the need for monitoring?
 Monitoring is a good management practice
 It helps managers plan, verify, and communicate what they aim to do, decide how to allocate resources,
learn how best to modify or redesign programs, and estimate the resulting program outputs, outcomes,
and impacts.
 Equip program managers with the information needed to improve their programs
 To communicate effectively to others the full range of benefits from R&D efforts
 It is a tool that not only helps measure a program’s success, but also contributes to its success.

IMPORTANCE OF MONITORING
R&D ACTIVITIES
Program managers may need to know:
 If their research is done right (efficiency and quality of research)
 If the program’s R&D efforts are focussed on the right research areas
 How program-created knowledge finds varied applications that generate additional benefits to the nation
 How collaborations and other activities stimulated by the program have affected nation’s R&D
capabilities.
 How their programs are providing benefits to the users of resulting energy-saving and energy-producing
innovations.
 How their programs are enhancing energy security by providing alternative energy sources, protecting
existing sources and having options ready for deployment if warranted by changing circumstances.
 If their past efforts were worth it and if planned new initiatives will be worth it.

WHAT IS MONITORING?
Definition: Monitoring is a continuous assessment of key program functions organized internally by
program management and carried out on an on-going basis. Monitoring entails setting up a data
collection system for compiling key data on program activities, participants, interim achievements and
outputs. The resulting data can be used to develop interim performance metrics or “indicators” of program
progress, outputs, and outcomes, and are helpful in keeping a program on track and for guiding mid-
course corrections. The data also contribute to evaluation studies.
Monitoring a program as it is carried out, collecting resulting data, and generating selected indicator
metrics from the data are integral to evaluation. Pairing monitoring with evaluation is considered good
practice. Continuous monitoring and data collection support evaluation and provide useful interim
indicators of change in key program functions that can guide program managers in making mid-course
corrections.
Limitations:
• Depends on appropriate selection of what is monitored.
• Interim indicators of progress are not measures of ultimate
achieved outcomes and impacts.
• Program often has multiple goals and it may be difficult to
know how multiple indicators inform the multiple goals.
Uses:
• To track interim program progress.
• To guide mid-course corrections; provide information to help program
managers make decisions to design or revise their program, re-direct
existing R&D funds, or allocate new funds.
• To support evaluation studies.

How monitoring and data collection are organized
and conducted:
Developing a monitoring system with data collection and construction of indicators starts with review of the
program’s detailed logic model.
From the logic model, it is possible to identify key activities, expected program participants, expected outputs, and,
perhaps, some expected outcomes that are conducive to monitoring, such as number of technologies under
commercialization.
A closer look at projects or research activities that comprise a program or initiative reveals the technical goals,
against which progress can be tracked.
After deciding what to monitor, the next step is to establish the supporting data collection strategies, databases,
and information technology framework.
It is necessary that program management identify which indicator metrics will best provide interim guidance.
Often graphical depictions of the selected indicators are helpful in revealing trends in key program functions, and
in guiding mid-course corrections.
When evaluation studies are launched, the data collected through program monitoring tend to be invaluable.

TECHNIQUES OF MONITORING
THE R&D ACTIVITIES:
Surveys Case study
Expert panels,
Peer review, &
Focus groups
Indicator
metrics
Bibliometrics
Historical
tracing
Econometrics Benchmarking
Network
analysis
Scorecard
Mission/
Outcome
mapping
Options Theory
Fore-sighting
Composite
Performance
Rating System
Cost-index
method
Market
Assessment

PEER REVIEW/ EXPERT JUDGEMENT
Definition: Peer Review/Expert Judgment is qualitative review, opinion, and advice from experts on the
subject being evaluated, based on objective criteria. The method combines program performance
information (provided to the experts) with the many years of cumulative experience of the subject-matter
experts, and focuses that informed expertise and experience on addressing key questions about a program,
initiative, project, proposal, paper, topic, or other subject of focus. While information from other sources,
including other methods of evaluation, may provide influential evidence, the ultimate conclusions about
performance are based on the judgment of the experts.
EERE’s Peer Review Guide (2004) defines in-progress peer review as:
A rigorous, formal, and documented evaluation process using objective criteria and qualified and
independent reviewers to make a judgment of the technical/ scientific/business merit, the actual or anticipated
results, and the productivity and management effectiveness of programs and/or projects.

How in-progress peer reviews are organized, conducted, and analyzed
• Peer Review Guide sets out minimum requirements for planning, conducting, and responding to peer
reviews.
• A primary requirement is that the reviews be independent both in fact and in terms of public
perception.
• This is achieved through having processes that are transparent and having third parties involved in the
selection of reviewers.
• To a large extent, the quality of the results depends upon the choice of qualified and independent
reviewers.
• In addition to being experts in the subject matter, reviewers should have no real or perceived conflict of
interest.
• Their judgments should be guided by the objective evaluation criteria, established prior to the review,
and should address the specific questions established for the review.
• When used to review an individual project or a collection of projects, peer review generally focuses on the
question “are we doing it right?” A program-level review will focus on the broader issue of “is the program
doing the right thing?”

• Used to answer a variety of questions throughout
the program performance cycle, as well as in other
applications.
• It is widely used by industry, government, and
academia. In practice, it ranges from a formal
process conducted according to strict protocol to
an informal process.
• It is used, for example, for support of strategic
planning decisions, selecting among projects and
programs, for in-progress project and program
review, for process assessment, for stage-gate
decisions, for merit review of papers for
publications, and for making judgments about
diverse topics, when supported by results from
application of other methods—the overall success
of a program.
Characteristics:
• Low-cost
• Fast-to-apply
• Widely accepted
• Versatile evaluation method
Uses:
• To conduct in-progress reviews of scientific quality
and productivity.
• To help answer questions about the relevancy,
timeliness, riskiness and management of existing
program research activities, and resource sufficiency
of new program initiatives.
• To score and rate projects under review to aid
decisions to continue, discontinue, or modify existing
or planned projects, programs, or program initiatives.
• To help assess appropriateness of program
mechanisms, processes, and activities and how they
might be strengthened.
• To integrate across multiple evaluation results and
render judgments about the overall success of a
program or program initiative.
Limitations:
• Steps may be needed to calibrate reviewer ratings
• Defining appropriate criteria
• Type of data needed for retrospective impact assessment cannot
be created in an expert review panel format

BIBLIOMETRIC METHODS-
COUNTS AND CITATION ANALYSIS
Definition: Counts of publications and patents are often used by R&D programs as indicators of program
knowledge outputs. Citation analysis of publications and patents is used to reveal relationships and linkages
between a program’s knowledge outputs and efforts undertaken by others. The frequency of citations may
signal the importance of a program’s knowledge outputs to others. Counts of publications and patents filed
are often included among an R&D program’s outputs, and measures of patents granted and citations of
publication and patents are often used in assessing an R&D program’s outcomes. Evaluators are increasingly
using patents and their citations as indicators of innovation, information flow, and value creation.
Bibliometric methods are used to show that knowledge has been created and disseminated, and to show
emergence of new ideas and development of relationships and patterns. These methods use text and text-
related materials to evaluate R&D programs. They are particularly relevant to R&D evaluation because the
output of research typically is knowledge, and knowledge is often expressed at least in part in reports,
publications, and patents.

How bibliometric studies of publication and patent counts and
citations are organized, conducted, and analyzed:
 Tabulating counts of publications and patents is relatively straightforward.
 These records are often routinely compiled by R&D organizations in their publication review and approval
and patent filing processes and captured in program performance monitoring systems.
 Search engines may also be helpful in compiling data on publications and in making comparisons.
 Citations analysis is generally performed as a forward search in time from an initial publication or patent
program output to downstream publications or patents which cite that produced by the program, but it
may also be performed backward to attribute current work to knowledge generated in the past by your
program or organization.
Uses:
• To provide measures of program knowledge outputs and evidence of outcome in the form of knowledge
dissemination and knowledge spillovers.
• To reveal linkages from Federal R&D to downstream outcomes.
• To identify users of a program’s knowledge and technology.

BIBLIOMETRIC METHODS- DATA MINING
Definition: Data mining is the extraction of key concepts or relationships from large quantities of digitized
natural language text. The method has also been called “literature-based discovery” (LBD), a descriptive
name. As in the case of the other bibliometric methods, this approach focuses on written documents--a
major output of research, and may include reports, publications, and textual components of patents or other
documents. Data mining enables the efficient and effective management and use of large volumes of R&D
texts by making it possible to integrate across document collections and to discover new information from
existing sources.
Data mining is a bibliometric method which searches texts for keywords to identify the origin of important
ideas and concepts. It is also used in evaluation to identify the emergence of relationships among research
organizations and disciplines.

How data mining studies are organized,
conducted, and analyzed:
 Data mining studies are organized to automate searches of large volumes of information in order to
identify relationships and patterns of interest that would be slow, and difficult or impossible to find by
human analysts working without specialized tools.
 Conducting a data mining study starts with the availability of subject texts in digital form; proceeds with
computer processing using various search and analysis algorithms to extract useful information, and
finishes with human analysis and interpretation of the results.
 Databases of technical reports, such as a program’s database of its own reports and the many others that
now exist (including, for example, Science Citation Index, Engineering Compendex, Medline, NTIS Technical
Reports, and RaDiUS database) can provide sources of text.
 Specialized data mining tools, developed and supplied by a variety of vendors, facilitate the processing
step; examples include WORDSTAT a data mining module for SIMSTAT, SPSS tools for data mining, SAS
data mining tools, among others.
 Information visualization support, such as that provided by PNNL, added to data mining serves as a
valuable assist in gaining insight from the results.

Uses:
 To show the origin of important ideas and concepts; more specifically to show that past investments in
R&D program contributed to emergent fields and technologies.
 To show relationships among research organizations and disciplines.
 To influence public S&T policy by providing decision makers insight into how cutting-edge technologies
develop from combinations of diverse research efforts over time.
 To gather technical intelligence that may alert evaluators to developments in precursor, rival, or
complementary technologies affecting the impact of a technology of interest.
 To provide information to help program managers make decisions to design or revise their program, re-
direct existing R&D funds, or allocate new funds.
 To help guide investment decisions in R&D in emergent areas.

BIBLIOMETRICS- HOTSPOT PATENT ANALYSIS
Definition: Hotspot patent analysis identifies patents that are highly cited by recently issued patents. The
technique offers an unobtrusive and unbiased way to uncover technological “hotspot clusters,” i.e., patented
technologies that are currently having a large impact on innovation. According to recent studies,
approximately 2% of recent patents are designated hotspots. Old patents can also be hotspots if there is a
recent big spike in citations of them.
A recently developed, specialized application of patent analysis in bibliometrics is “Hotspot Patent Analysis,”
which looks at patenting frequency to identify patents that appear to be having a particularly large impact
on innovation and also “Next Generation Patents,” which are building on “Hotspot Clusters.” This analysis
helps to assess the relative importance of a program’s patents to technological innovation.
Limitations:
 Issues with completeness and accuracy of data records
 Interpretation of results and their implications for different R&D programs

Uses:
 To identify current clusters of intensive innovative activity and developing “hot” trends in technology.
 To assess the positioning of an R&D program’s output (as measured by patents and citations of the
agency’s publications by patents) relative to new and developing clusters of innovative activity.
 To identify the regional impact of a public R&D program in order to better organize the program’s
outreach activities.
 To analyze the organizational and collaborative characteristics of identified clusters of innovative activity.
 To analyze hotspots in a selected technology area (e.g., fuels, alternative vehicles, etc.) in order to assess
how these hotspots and next generation patents are linked to an R&D program (e.g., EERE).
 To gather competitive technical intelligence for R&D organizations.

SURVEY METHOD
Definition: Survey is a method of obtaining information directly from people about their ideas, opinions,
attitudes, beliefs, preferences, concerns, plans, experiences, observations, and virtually any other issue. A
survey collects information by asking people questions and recording their responses. Surveys are often
used when the desired data are not available through other sources, but could be obtained by asking
people. Surveys are used in R&D evaluation for a variety of purposes, such as learning about a program’s
progress and effects; discovering the opinions of those participating in a program or using its outputs;
addressing stakeholder questions; and gathering information to supplement other sources of information.
A survey collects information by asking people questions—the answers to which can be expressed in terms
of statistics. This method is particularly useful for characterizing a program’s progress, learning more detailed
information about that progress, assessing customer satisfaction, and answering a variety of stakeholder
questions

How survey studies are organized,
Given that surveys obtain information from people, it is necessary to decide which people and how many people
to ask; what to ask; how to structure the questions; when and how often to ask; and how to submit the
questions—all aspects of survey design.
Using samples reduces study costs, but if a sample is too small, it may not adequately represent the population
of people it is intended to represent.
A survey may be administered just once to a given group to obtain data as they exist at that time, in which case it
is “cross-sectional.” Alternatively, a survey may be administered to the same group of people at different times
to assess changes over time within the group, in which case it is “longitudinal.”
Alternatives for administering a survey include paper copies or computer disks, electronic copies sent via e-mail
or web-based, questions asked by an interviewer face-to-face or by telephone.
A balance must be struck between collecting needed data and avoiding being overly intrusive or imposing too
much burden. Survey data are commonly analyzed using descriptive statistics, such as counts, percentages,
averages, ranges, and measures of central tendency and measures of variation. Survey data may also be analyzed
to show relationships, to compare groups, and to determine trends and changes over time.

Uses:
 To describe a program statistically.
 To assess customer satisfaction.
 To answer stakeholder questions about a program and its effects.
 To support evaluation studies.
Limitations:
 Failure to adequately reflect the target population (due to weakness in survey design)
 Individual responses are understood to be treated as confidential without the explicit agreement of the
respondent to the contrary.

TECHNOLOGY COMMERCIALIZATION
TRACKING MODEL
Definition: The technology commercialization tracking method tracks the new, energy-efficiency technologies
developed through R&D projects sponsored by the program and that may include research cost-shared with
industry. It classifies those technologies showing a requisite level of development, such as either “emerging,”
“commercially successful,” or “mature.” For example, a technology could be considered emerging when it is
thought to be within approximately one to three years of commercialization. It could be considered
commercially successful when full-scale commercial units of a technology have been made operational in
private industry and are available for sale. It could be considered mature once a commercially successful
technology has been in operation for 10 years or longer. It is considered a historical technology when it is no
longer being sold in the U.S. When a technology is emerging, preliminary information is collected. When a
technology is commercially successful, it is placed on the active tracking list and additional data are collected,
which are used to analyze benefits from program-sponsored R&D. Mature and historical technologies do not
need to be tracked.

Technology commercialization tracking involves monitoring technologies considered to be commercially
successful and their associated energy savings, economic and environmental benefits. Market and cost data is
used to estimate the cumulative net benefits of the program.
Uses:
 To identify which projects funded by the program were commercialized and to what extent.
 To provide documented evidence of the impact of program-sponsored technology development and
deployment efforts.
 To estimate the cumulative net benefits of the program.

NETWORK ANALYSIS
Definition: This is a method of visually mapping and measuring relationships and linkages among
researchers, groups of researchers, laboratories, or other organizations. Network analysis is relevant to
evaluation of R&D programs because it identifies routes of interactions by which ideas, knowledge, and
information flow among participants in R&D, thereby possibly influencing the nature, quality, quantity, and
speed of research and innovation, as well as the dissemination of created knowledge through the network.
The underlying concept is that the conduct of science is a social activity collectively performed and
resulting in “communities of practice.” Advances in knowledge stem from knowledge sharing and
knowledge combining activities. Networks can link researchers to a rich flow of ideas and information. A
network of researchers creates a knowledge system that may yield much more than the individuals acting
independently. The network analysis method, which examines flows of knowledge into and through the
social network, is seen as a promising approach to understanding, predicting, and improving knowledge
outcomes. Network shape, size, and density can serve as indicators of the strength of communities of
practice and signal relative roles and relationships.

Limitations:
 It does not provide a quantitative measure of its value.
 Network diagram may be time limited
 Costs can be a limiting factor in the use of network analysis
Network analysis, which shows linkages among researchers or organizations and how they develop over
time, is useful in assessing a program’s impact on collaboration and emerging roles and positions of
influence of researchers and organizations. While bibliometric methods show how knowledge is
disseminated via citing publications and patents, network analysis shows how knowledge particularly tacit
knowledge is disseminated via a variety of communication flows. Development of research networks is
significant because it is expected to increase research capabilities, progress, and impacts.

How network analysis studies are organized,
To analyze results of network analysis, several measures are used.
1. One measure is of centrality, based on the number of direct links one node has to other nodes.
2. Another measure indicates the extent of influence a node has over flows in the network to other nodes.
3. A third measure indicates how closely linked a nodule is both directly and indirectly to all other nodules in
the network.
An entity in a network may be a “hub,” i.e., a node with a high degree of centrality and also important as a link to
other parts of the network. An entity may be a “peripheral player,” a node on the periphery of an identified
network. It may be a “boundary spanner,” a node connecting one network cluster to another.
“Clusters” may develop within networks, i.e., groups within the network connected through multiple links.
Analysis of networks can reveal areas of potential failure, such as the vulnerability of a highly centralized
network to the departure of a key researcher. It can also reveal areas of strength, such as clusters with connection
redundancies that make them less vulnerable to removal of single links or nodes

Uses:
 To analyze the impact of R&D policies on collaborative activity.
 To reveal dominant researchers or research organizations, and/or to assess the openness of networks to
members.
 To improve understanding of how and why collaborations develop, what form they take, and their
 To investigate and demonstrate the impact of an R&D program on applications by examining the flow of
knowledge among researchers, groups of researchers, and users.
 To identify and foster emergent knowledge systems; to assess their strengths and weaknesses.
 To highlight the importance to participants of intangible asset development, and to assess more fully
knowledge spillovers.

CASE STUDY METHOD
Definition: The case study method presents information through a narrative about the subject, often with
supporting data displayed in tables and graphs. Case study is a method widely used by R&D programs for
evaluation, both to describe programs and how they work and to investigate underlying functional
relationships. One reason for its widespread use is that the narrative can often capture the richness of detail
and complexities of scientific research and development, particularly for a non-scientific audience, while
quantitative results can be often be provided with little additional effort.
Case studies use narratives supported by data to describe, explain, and explore phenomena and events.
Case studies are a particularly useful strategy for addressing how and why questions within a real-life
context. For example, case studies can be used to shed light on how innovation occurs, why certain
decisions are made, and why some processes work better than others.

How case studies are organized, conducted,
and analyzed:
There is no single format, rather there are a variety of formats or structures which may be used for writing
case-studies.
One format is the chronological structure, reflecting cases that cover events as sequences over time.
Descriptive case studies are often presented using key areas of information as subtopics, such as the origin
of a research idea, the source of technology, the role of government, estimated sales, etc.
Exploratory and explanatory case studies may present a series of hypotheses, followed by what the case or
cases show.
A linear-analytical structure is one of several other alternatives, whereby the purpose of the study is
followed by the methodology, the findings from information collected and analyzed, and then by conclusions.

Uses:
 To explore the genesis of research ideas and their consequences.
 To tell the stories of the people, organizations, projects, and programs involved in scientific pursuit.
 To investigate underlying theories and explore process dynamics.
 To answer specific what, why, and how questions.
 To provide illustrative examples of how a program works.
Limitations:
• Provides anecdotal evidence rather than quantitative evidence
• Less persuasive

THERE ARE MANY MORE TOOLS:
1. Econometric methods encompass a number of mathematical and statistical techniques that are used to
increase the rigor of estimation of economic relationships. Econometric methods are often used to
estimate program impacts.
2. The historical tracing method has been successfully used to trace highly successful commercial products
back to earlier DOE research. Showing these linkages helps to demonstrate the importance of past
research and suggests the potential importance of present research not yet incorporated in commercial
products.
3. Benchmarking means making comparisons to see how people, programs, organizations, regions, or
countries compare in terms of some aspect of their performance—to identify where and how to make
improvements.

4. Benefit-cost case studies quantify positive and negative effects of a project, a cluster of projects, or a
program, and compare benefits against the costs using any of several measures. An essential feature of
this method is accounting for “additionality,” i.e., benefits and costs with the project, cluster of projects, or
program as compared with the benefits and costs without it. Benefit-cost analysis has a long history of
use in evaluating government projects.
5. Spillover analysis can be used to measure the surplus benefits to producers who use new and
improved technologies, the surplus benefits to consumers who buy goods and services incorporating the
new and improved technologies, the benefits to those in other industries who are able to use the
knowledge from the research without having paid for it, and the benefits realized by those whose
existing goods and services are increased in value due to complementarities of the new and improved
technologies. Spillover analysis reveals to a fuller extent the value to society of research and thereby is
important to avoiding underinvestment in research.

CONCLUSION
 There is no single tool or technique that can handle the whole process to simultaneously shape, integrate
and monitor R&D strategy to support corporate strategy.
 Peer review/expert judgment helps R&D managers learn how to design and redesign program elements
and processes, to select projects, to decide whether to continue or discontinue projects, and how best to
modify the research direction of the R&D portfolio.
 Network analysis is useful for answering questions about a program’s impact on collaborative research
and the dissemination of knowledge particularly tacit knowledge.
 Surveys are useful in answering a host of questions, such as how satisfied are the program’s customers
and how are customers using program outputs.
 Citation analysis helps document the linkages between a program’s outputs and downstream users.

 Economic case studies can estimate the benefits and costs of program outputs, including those
measurable in monetary terms and those more difficult to measure such as environmental effects and
energy security effects.
 Benchmarking can help identify where and how to make improvements by comparing a program with its
counterparts abroad.
 Econometric methods can help demonstrate that it was the program that caused an outcome and not
something else.
 Using these and other methods can help a program manager better understand and manage his or her
program so as to achieve its goals, and obtain results needed to communicate achievements to others.
 There is a need to join the best tools and techniques in a literature that can provide managers with an
overview of how to shape, integrate and monitor R&D strategy to successfully support strategy.

Monitoring R&D functions

Recommended

More Related Content

What's hot (20)

Similar to Monitoring R&D functions (20)

Recently uploaded (20)

Monitoring R&D functions

Editor's Notes