Contemporary Empirical Methods In Software

Engineering 1st Ed Michael Felderer download

https://ebookbell.com/product/contemporary-empirical-methods-in-
software-engineering-1st-ed-michael-felderer-22417778

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Contemporary Empirical Political Theory Reprint 2020 Kristen Renwick

Monroe Editor

https://ebookbell.com/product/contemporary-empirical-political-theory-
reprint-2020-kristen-renwick-monroe-editor-51815914

Industrial Organization Contemporary Theory And Empirical Applications

5th Edition Lynne Pepall

https://ebookbell.com/product/industrial-organization-contemporary-
theory-and-empirical-applications-5th-edition-lynne-pepall-6826654

The Protection Of General Interests In Contemporary International Law

A Theoretical And Empirical Inquiry Isn9780192846501 Massimo Iovane
Editor

https://ebookbell.com/product/the-protection-of-general-interests-in-
contemporary-international-law-a-theoretical-and-empirical-inquiry-
isn9780192846501-massimo-iovane-editor-36128252

The Meaning Of Citizenship In Contemporary Chinese Society An

Empirical Study Through Western Lens 1st Edition Sicong Chen Auth

https://ebookbell.com/product/the-meaning-of-citizenship-in-
contemporary-chinese-society-an-empirical-study-through-western-
lens-1st-edition-sicong-chen-auth-6793288
From Psychology To Phenomenology Franz Brentanos Psychology From An
Empirical Standpoint And Contemporary Philosophy Of Mind Biagio G
Tassone Auth

https://ebookbell.com/product/from-psychology-to-phenomenology-franz-
brentanos-psychology-from-an-empirical-standpoint-and-contemporary-
philosophy-of-mind-biagio-g-tassone-auth-5373942

Compliance With Planning Standards Related To The Setbacks Around

Domestic Buildings Empirical Evidence From Kenya 2nd Edition Dr
Wilfred Ochieng Omollo

https://ebookbell.com/product/compliance-with-planning-standards-
related-to-the-setbacks-around-domestic-buildings-empirical-evidence-
from-kenya-2nd-edition-dr-wilfred-ochieng-omollo-22982660

Agricultural Transformation In Africa Contemporary Issues Empirics And

Policies Gbadebo O A Odularu

https://ebookbell.com/product/agricultural-transformation-in-africa-
contemporary-issues-empirics-and-policies-gbadebo-o-a-odularu-48813788

Contemporary Science Education And Challenges In The Present Society

Perspectives In Physics Teaching And Learning Maurcio Pietrocola Iv
Gurgel Cristina Leite

https://ebookbell.com/product/contemporary-science-education-and-
challenges-in-the-present-society-perspectives-in-physics-teaching-
and-learning-maurcio-pietrocola-iv-gurgel-cristina-leite-44942648

Contemporary Skull Base Surgery A Comprehensive Guide To Functional

Preservation A Samy Youssef Editor

https://ebookbell.com/product/contemporary-skull-base-surgery-a-
comprehensive-guide-to-functional-preservation-a-samy-youssef-
editor-44944112
Michael Felderer
Guilherme Horta Travassos Editors

Contemporary
Empirical Methods
in Software
Engineering
Contemporary Empirical Methods in Software
Engineering
Michael Felderer • Guilherme Horta Travassos
Editors

Contemporary
Empirical Methods in
Software Engineering
Editors
Michael Felderer Guilherme Horta Travassos
Department of Computer Science Department of Systems Engineering and
University of Innsbruck Computer Science, COPPE
Innsbruck, Austria Federal University of Rio de Janeiro
Rio de Janeiro, Brazil

ISBN 978-3-030-32488-9 ISBN 978-3-030-32489-6 (eBook)

https://doi.org/10.1007/978-3-030-32489-6

© Springer Nature Switzerland AG 2020

Chapter 17 is licensed under the terms of the Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/). For further details see licence information in the chapter.
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Foreword

As the name of the field suggests, software engineering is expected to be an engi-

neering discipline. However, it is not governed, to the same extent, by underlying
mathematical models as many other engineering disciplines, in particular, those
addressing physical artifacts as in electrical engineering or mechanical engineering.
Thus, mathematics is insufficient to conduct research and improve in software
engineering, although it is vital for some sub-areas within software engineering.
There are several reasons for this insufficiency.
First of all, the software is invisible (Brooks 1987). We can read the code, but
we cannot see it in use. We can only observe the effect of the software being
executed. Furthermore, software engineering is intrinsically complex since it is,
to a considerable extent, dependent on the knowledge and capability of humans
developing the software. Moreover, the ability of the individuals to work in a
team contributing to the same software system is essential. The development is
supported by different processes, methods, techniques, languages, and tools, which,
in one way or another, are used by the organization developing the software. Thus,
software engineering is an interplay between human knowledge, social networks
of the individuals, and available assets in the organization developing the software
(Wohlin et al. 2015).
To be able to study and improve the way software is engineered, many
researchers have embraced and promoted software engineering as an empirical
engineering discipline. Empirical studies were conducted early in the discipline,
but they were quite rare. In 1986, an article describing experimentation in software
engineering was published (Basili et al. 1986) outlining software engineering as
an experimental science. The establishment of empirical software engineering was
done to a large extent in the 1990s. At the beginning of the twenty-first century,
two books on experimentation in software engineering were published (Wohlin et
al. 2012; Juristo and Moreno 2001). The former book came in a second edition in
2012 (Wohlin et al. 2012), and it was published in Chinese in 2015.
In 2004, the concept of evidence-based software engineering was established
in software engineering (Kitchenham et al. 2004). The evidence is most often
generated from empirical studies, and hence, it was a natural continuation of the

v
vi Foreword

previous work on empirical software engineering. As the area of empirical software

engineering became well established, the need for advances in our conduct of
empirical studies grew (Shull et al. 2008). Given the applied nature of software
engineering, the need to conduct empirical studies in a real-life context was
strengthened by the publication of guidelines for conducting case studies (Runeson
et al. 2012).
As a continuation concerning the focus on evidence in software engineering, a
book on evidence-based software engineering was published in 2015 (Kitchenham
et al. 2015). Furthermore, empirical software engineering has gone from being a
sub-area of software engineering to be an integral part of software engineering.
Nowadays, it is expected that research is evaluated and assessed using empirical
methods. Thus, it is, in most cases, insufficient to present an idea or a solution
without empirical evidence. In summary, software engineering has moved into truly
being an engineering discipline.
The book Contemporary Empirical Methods in Software Engineering, edited
by Prof. Michael Felderer and Prof. Guilherme Horta Travassos, takes the next
step by including chapters on essential and timely topics in empirical software
engineering. The chapters are written by some of the world’s leading experts on
empirical methods in software engineering. The editors have done an excellent job
of attracting experts in the field who contribute with essential topics concerning the
empirical software engineering of today.
The book follows up on the previous books and articles on empirical and
evidence-based software engineering. As the title of the book suggests, the book
takes a timely step in including a set of chapters addressing emerging areas in
empirical software engineering. It provides an excellent combination of chapters
addressing contemporary areas of interest for anyone conducting research in
software engineering and in particular for those with a strong focus on empirical
software engineering. The book is a highly recommended read for, in particular,
Ph.D. students and researchers interested in conducting high-quality software
engineering research aspiring to apply empirical research methods for today and
the future.

Blekinge Institute of Technology Claes Wohlin

Karlskrona, Sweden

References

Basili VR, Selby RW, Hutchens DH (1986) Experimentation in software engineer-

ing. IEEE Trans Softw Eng SE-12(7):733–743
Brooks FP Jr (1987) No silver bullet – essence and accidents of software engineer-
ing. IEEE Comput 20(4):10–19
Juristo N, Moreno AM (2001) Basics of software engineering experimentation.
Springer, New York
Foreword vii

Kitchenham BA, Dybå T, Jørgensen M (2004) Evidence-based software engineer-

ing. In: Proceedings of 26th international conference on software engineering,
Edinburgh, pp 273–281
Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineer-
ing and systematic reviews. Chapman and Hall/CRC, Boca Raton
Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software
engineering – guidelines and examples. Wiley, Hoboken
Shull F, Singer J, Sjøberg DIK (eds) (2008) Guide to advanced empirical software
engineering. Springer, London
Wohlin C, Runeson P, Höst M, Regnell B, Ohlsson MC, Wesslén A (2012)
Experimentation in software engineering. Springer, Berlin
Wohlin C, Šmite D, Moe NB (2015) A general theory of software engineering:
balancing human, social and organizational capitals. J Syst Softw 109:229–242
Contents

The Evolution of Empirical Methods in Software Engineering .. . . . . . . . . . . . 1

Michael Felderer and Guilherme Horta Travassos

Part I Study Strategies

Guidelines for Conducting Software Engineering Research . . . . . . . . . . . . . . . . 27
Klaas-Jan Stol and Brian Fitzgerald
Guidelines for Case Survey Research in Software Engineering . . . . . . . . . . . . 63
Kai Petersen
Challenges in Survey Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 93
Stefan Wagner, Daniel Mendez, Michael Felderer, Daniel Graziotin,
and Marcos Kalinowski
The Design Science Paradigm as a Frame for Empirical Software
Engineering .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 127
Per Runeson, Emelie Engström, and Margaret-Anne Storey

Part II Data Collection, Production, and Analysis

Biometric Measurement in Software Engineering . . . . . . .. . . . . . . . . . . . . . . . . . . . 151
Fabian Fagerholm and Thomas Fritz
Empirical Software Engineering Experimentation with Human
Computation .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 173
Marta Sabou, Dietmar Winkler, and Stefan Biffl
Data Science and Empirical Software Engineering . . . . . .. . . . . . . . . . . . . . . . . . . . 217
Ezequiel Scott, Fredrik Milani, and Dietmar Pfahl
Optimization in Software Engineering: A Pragmatic Approach . . . . . . . . . . . 235
Günther Ruhe

ix
x Contents

The Role of Simulation-Based Studies in Software Engineering

Research .. . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 263
Breno Bernard Nicolau de França and Nauman Bin Ali
Bayesian Data Analysis in Empirical Software Engineering:
The Case of Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 289
Richard Torkar, Robert Feldt, and Carlo A. Furia

Part III Knowledge Acquisition and Aggregation

Automating Systematic Literature Review . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 327
Katia R. Felizardo and Jeffrey C. Carver
Rapid Reviews in Software Engineering. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 357
Bruno Cartaxo, Gustavo Pinto, and Sergio Soares
Benefitting from the Grey Literature in Software Engineering
Research .. . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 385
Vahid Garousi, Michael Felderer, Mika V. Mäntylä, and Austen Rainer
Guidelines for Managing Threats to Validity of Secondary Studies
in Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 415
Apostolos Ampatzoglou, Stamatia Bibi, Paris Avgeriou,
and Alexander Chatzigeorgiou
Research Synthesis in Software Engineering . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 443
Paulo Sérgio Medeiros dos Santos and Guilherme Horta Travassos

Part IV Knowledge Transfer

Open Science in Software Engineering . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 477
Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold
Third Generation Industrial Co-production in Software Engineering. . . . . 503
Tony Gorschek and Krzysztof Wnuk
The Evolution of Empirical Methods
in Software Engineering

Michael Felderer and Guilherme Horta Travassos

Abstract Empirical methods like experimentation have become a powerful means

to drive the field of software engineering by creating scientific evidence on software
development, operation, and maintenance, but also by supporting practitioners in
their decision-making and learning. Today empirical methods are fully applied in
software engineering. However, they have developed in several iterations since the
1960s. In this chapter we tell the history of empirical software engineering and
present the evolution of empirical methods in software engineering in five iterations,
i.e., (1) mid-1960s to mid-1970s, (2) mid-1970s to mid-1980s, (3) mid-1980s to end
of the 1990s, (4) the 2000s, and (5) the 2010s. We present the five iterations of
the development of empirical software engineering mainly from a methodological
perspective and additionally take key papers, venues, and books, which are covered
in chronological order in a separate section on recommended further readings, into
account. We complement our presentation of the evolution of empirical software
engineering by presenting the current situation and an outlook in Sect. 4 and the
available books on empirical software engineering. Furthermore, based on the
chapters covered in this book we discuss trends on contemporary empirical methods
in software engineering related to the plurality of research methods, human factors,
data collection and processing, aggregation and synthesis of evidence, and impact
of software engineering research.

Guilherme Horta Travassos is a CNPq Researcher.

M. Felderer ()
Department of Computer Science, University of Innsbruck, Innsbruck, Austria
Department of Software Engineering, Blekinge Institute of Technology, Karlskrona, Sweden
e-mail: [email protected]
G. H. Travassos
Department of Systems Engineering and Computer Science, COPPE, Federal University of Rio
de Janeiro, Rio de Janeiro, Brazil
e-mail: [email protected]

© Springer Nature Switzerland AG 2020 1

M. Felderer, G. H. Travassos (eds.), Contemporary Empirical Methods in Software
Engineering, https://doi.org/10.1007/978-3-030-32489-6_1
2 M. Felderer and G. H. Travassos

1 Introduction

The term software engineering originated in the early 1960s (Hey et al. 2014).
During the NATO Software Engineering Conferences held in 1968 and 1969,
participants made explicit that engineering software requires dedicated approaches
that are separate from those for the underlying hardware systems. Until that
“software crisis,” software-related research mostly focused on theoretical aspects,
e.g., algorithms and data structures used to write software systems, or practical
aspects, e.g., an efficient compilation of software for particular hardware sys-
tems (Guéhéneuc and Khomh 2019). Since then, these topics are investigated in
computer science, which pertains to understanding and proposing theories and
methods related to the efficient computation of algorithms, and differs from software
engineering (research), which has become a very dynamic discipline on its own
since its foundation in the 1960s. IEEE (1990, 2010) defines software engineering
(SE) as: (1) The application of a systematic, disciplined, quantifiable approach to
the development, operation, and maintenance of software, that is, the application
of engineering to software, and (2) The study of approaches as in (1). Software
engineering also differs from other engineering disciplines due to the immaterial
nature of software not obeying physical laws and the importance of human factors
as software is written by people for people. Software engineering is fundamentally
an empirical discipline, where knowledge is gained applying direct and indirect
observation or experience. Approaches to software development, operation, and
maintenance must be investigated by empirical means to be understood, evaluated,
and deployed in proper contexts. Empirical methods like experimentation are
therefore essential in software engineering to gain scientific evidence on software
development, operation, and maintenance, but also to support practitioners in their
decision-making and learning (Travassos et al. 2008). The application of empirical
methods makes software engineering more objective and less imprecise, facilitating
the transfer of software technologies to the industry (Shull et al. 2001). Software
engineers learn by observing, exploring, and experimenting. The level of learning
depends on the degree of observation or intervention (Thomke 2003) promoted by
the experiences and studies performed.
Traditionally, empirical software engineering (ESE) is the area of research that
emphasizes the use of empirical methods in the field of software engineering.
According to Harrison and Basili (1996), “Empirical software engineering is the
study of software-related artifacts for the characterization, understanding, eval-
uation, prediction, control, management, or improvement through qualitative or
quantitative analysis. The quantitative studies may range from controlled experi-
mentation to case studies. Qualitative studies should be well-defined and rigorous.”
The role and importance of the different types of empirical methods in software
engineering have evolved since the foundation of software engineering. In this
chapter, we discuss the evolution of empirical methods in software engineering and
especially also take key venues and books into account as they reflect that evolution.
The Evolution of Empirical Methods in Software Engineering 3

The chapter is organized as follows: In Sect. 2, we provide background on

empirical research methods in software engineering. In Sect. 3, we present the
evolution of empirical software engineering by describing five iterations of its devel-
opment. Based on that “historical” perspective on empirical software engineering,
in Sect. 4 we describe current trends in empirical software engineering based on the
chapters on contemporary empirical methods in software engineering covered in this
book. In Sect. 5, we present the available books on empirical methods in software
engineering in chronological order as recommended further reading. Finally, in
Sect. 6, we conclude this chapter.

2 Empirical Research Methods in Software Engineering

The scientific approach typically consists of observation, measurement, and exper-

imentation. Observation helps researchers to formulate essential questions about
a phenomenon under study to build models and to derive hypotheses that can
be tested through experimentation. Measurement is essential for both observation
and experimentation. A scientific hypothesis must be refutable to be meaningfully
tested. Tested hypotheses are compiled and communicated in the form of laws or
theories. At the heart of the scientific approach are research methods in general and
the empirical method in particular. Empirical methods leverage evidence obtained
through observation, measurement, or experimentation to address a scientific
problem. Evidence should be based on qualitative and quantitative research. In this
section, we provide an overview of research methods in software engineering in
general and empirical methods in particular.

2.1 Research Methods

To perform scientific research in software engineering, one has to understand

the available research methods and their limitations. For the field of software
engineering, Basili (1993) and Glass (1994) summarized four research methods:
scientific, engineering, empirical, and analytical.
The so-called scientific method observes the world and builds a model based on
the observations, e.g., a simulation model of the software process or product. The
scientific method is inductive and tries to extract from the world some model that
can explain a phenomenon and to evaluate whether the model is representative for
the phenomenon under observation. It is a model-building approach.
The engineering method studies current solutions, proposes changes, and then
evaluates them. It suggests the most appropriate solutions, develops, measures and
analyzes, and repeats until no further improvement is possible. It is an evolutionary
improvement-oriented approach that assumes the existence of some model of the
software process or product. It modifies this model to improve the objects of study.
4 M. Felderer and G. H. Travassos

The empirical method proposes a model and evaluates it through empirical

studies like case studies or experiments. The empirical method normally follows
an iterative and incremental approach that can begin with an exploratory survey,
followed by case studies in an industrial context to better understand specific
phenomena and controlled experiments to investigate cause–effect relationships.
The analytical method proposes a formal theory, develops the theory, derives the
results, and, if possible, compares it with empirical observations. It is deductive and
provides an analytical basis for developing a model.
Traditionally, the analytical method is used in the more formal areas of electrical
engineering and computer science, but is important for software engineering as
well, e.g., when building mathematical models for software reliability growth (Lyu
et al. 1996). The scientific method, inspired by natural science, is traditionally
used in applied areas, such as the simulation of a sensors network to evaluate
its performance. However, simulations are used as a means for conducting an
experiment as well (Baros et al. 2004). The engineering method is dominating
in industry (Wohlin et al. 2012). The empirical method, mainly using empirical
strategies, has traditionally been used in social sciences and psychology, where
one is unable to state any laws of nature but concerned with human behavior. The
engineering and the empirical method can be seen as variations of the scientific
method (Basili 1993). This overlap and an integrated view of the scientific, engineer-
ing, and empirical methods is also an underlying design principle of this book on
empirical methods. It considers not only chapters on traditional empirical strategies
like surveys (see chapter “Challenges in Survey Research”), but for instance, also a
chapter on simulation-based studies (see chapter “The Role of Simulation-Based
Studies in Software Engineering Research”), which are closer to the scientific
method as defined above, or a chapter on design science (see chapter “The Design
Science Paradigm as a Frame for Empirical Software Engineering”), which can
tightly be linked to the engineering method. All of these investigation strategies
refer to empirical methods.

2.2 Empirical Methods

Empirical methods rely on the collected data. Data collection methods may involve
qualitative or quantitative data. Some widely used qualitative data collection
methods in software engineering are interviews and participant observation (Seaman
1999). Some commonly used quantitative data collection methods are archival data,
surveys, experiments, and simulation (Wohlin et al. 2012). Once data are collected,
the researcher needs to analyze the data by using qualitative analysis methods,
e.g., grounded theory, thematic analysis, or hermeneutics, and quantitative analysis
methods, e.g., statistical analysis and mathematical modeling approaches.
In general, there are three widely recognized research processes called quanti-
tative research, qualitative research, and semiquantitative research. An alternative
option is the combination of both qualitative and quantitative research, denoted as
The Evolution of Empirical Methods in Software Engineering 5

mixed research (Creswell and Creswell 2018). The distinction between qualitative
and quantitative research comes not only from the type of data collected, but also
the objectives, types of research questions posed, analysis methods, and the degree
of flexibility built into the research design as well (Wohlin and Aurum 2015).
Qualitative research aims to understand the reason (i.e., “why”) and mechanisms
(i.e., “how”) explaining a phenomenon. A popular method of qualitative research
is case study research, which examines a set of selected samples in detail to
understand the phenomenon illustrated by the samples. For instance, a qualitative
study can be conducted to understand the impediments of automating system
tests. Quantitative research is a data-driven approach used to gain insights about
an observable phenomenon. Data collected from observations are analyzed using
mathematical and statistical models to derive quantitative relationships between
different variables capturing different aspects of the phenomenon under study. A
popular method of quantitative research are controlled experiments to examine
cause–effect relationships between different variables characterizing a phenomenon
in a controlled environment. For instance, different review techniques could be
compared via a controlled experiment. Mixed research collects quantitative and
qualitative data. It is a particular form of multi-method research, which combines
different research methods to answer some hypotheses, and is often used in empir-
ical software engineering due to the lack of theories in software engineering with
which we interpret quantitative data and due to the need to discuss qualitatively the
impact of the human factor on any experiments in software engineering (Guéhéneuc
and Khomh 2019). Semiquantitative research deals with approximate measurements
to data rather than exact measurements (Bertin 1978). It looks for understanding the
behavior of a system based on causal relations between the variables describing
the system. Semiquantitative models allow one to express what is known without
making inappropriate assumptions, simulating ranges of behavior rather than values
of point (Widman 1989). It has many applications in both the natural and social
sciences. Semiquantitative research supports cases where direct measurements are
not possible, but where it is possible to estimate an approximated behavior. In other
words, this type of study is applied in scenarios where the numerical values in the
mathematical relations governing the changes of a system are not known. In this
context, the direction of change is known, but not the size of its effect (Ogborn and
Miller 1994). Simulation-based studies in software engineering can benefit from
using semiquantitative research (Araújo et al. 2012).
The three major and well-established empirical methods in software engineering
are: survey, case study, and experiment (Wohlin et al. 2012). Primary studies using
such methods can be performed in vivo, in vitro, in virtuo, and in silico (Travas-
sos and Barros 2003). In vivo studies involve participants and projects in their
natural environments and contexts. Such studies are usually executed in software
development organizations throughout the software development process under real
working conditions. In vitro studies are performed in controlled environments, such
as laboratories or controlled communities, under configured working conditions. In
virtuo studies have the subjects interacting with a computerized model of reality.
The behavior of the environment with which subjects interact is described as a
6 M. Felderer and G. H. Travassos

model and represented by a computer program. In silico studies represent both

subjects and real world as computer models. The environment is fully composed
of computer models to which human interaction is reduced to a minimum.
A survey is a system for collecting information from or about subjects (people,
projects, among others) to describe, compare, or explain their knowledge, attitudes,
and behavior (Fink 2003). A survey is often an investigation performed in retrospect,
when, for instance, a tool or technique has been in use for a while (Pfleeger 1995).
The primary means of gathering qualitative or quantitative data are interviews or
questionnaires. These are done through taking a sample that is representative of the
population to which is generalized.
A case study in software engineering is an empirical inquiry that draws on
multiple sources of evidence to investigate one or a small number of instances
of a contemporary software engineering phenomenon within its real-life context,
especially when the boundary between phenomenon and context cannot be clearly
specified (Runeson et al. 2012).
An experiment is used to examine cause–effect relationships between different
variables characterizing a phenomenon (Guéhéneuc and Khomh 2019). Experi-
ments allow researchers to verify, refute, or validate hypotheses formulated about
the phenomenon under study. In a controlled experiment, one variable of the study
setting is manipulated, and based on randomization, different treatments are applied
to or by different subjects while keeping other variables constant, and measuring
the effects on outcome variables (Wohlin et al. 2012). A quasi-experiment is similar
to a controlled experiment, where the assignment of treatments to subjects cannot
be based on randomization, but emerges from the characteristics of the subjects
or objects themselves (Wohlin et al. 2012). Replication experiments reproduce
or quasi-reproduce previous experiments with the objectives to confirm or infirm
the results from previous experiments or to contrast previous results in different
contexts (Guéhéneuc and Khomh 2019).
Regardless of the applied empirical method, to acquire scientific evidence about
the investigated software engineering phenomena involves observation, measure-
ment, and experimentation of the world and existing solutions. It demands the
proposition of models and theories describing the observed behavior, collecting
and analyzing data, putting the hypotheses under proof, and repeating the overall
process over time to strengthen the evidence on the observed phenomena. Based on
several primary studies, in which direct observations and measurements about the
objects of interest are made, whether by surveys, experiments, or case studies, which
are there also called empirical strategies, one can perform secondary studies. A
secondary study does not generate any data from direct observation or measurement,
instead, it analyzes a set of primary studies and usually seeks to aggregate the
results from these to provide stronger forms of evidence about a particular phe-
nomenon (Kitchenham et al. 2015). Secondary studies typically appear as systematic
(literature) reviews, which aim to provide an objective and unbiased approach to
finding relevant primary studies, and for extracting, aggregating, and synthesizing
the data from these (Kitchenham et al. 2015). A particular type of a systematic
review is a systematic mapping study (Petersen et al. 2015), which classifies studies
The Evolution of Empirical Methods in Software Engineering 7

to identify clusters of studies (that could form the basis of a fuller review with more
synthesis) and gaps indicating the need for more primary studies.
The scientific or industrial significance of empirical studies depends on their
validity, i.e., the degree to which one can trust the outcomes of an empirical
study (Kitchenham et al. 2015). Validity is usually assessed in terms of four
commonly encountered forms of threats to validity: internal, external, construct,
and conclusion validity (Shadish et al. 2002). Internal validity refers to inferences
that the observed relationship between treatment and outcome reflects a cause–
effect relationship. External validity refers to whether a cause–effect relationship
holds over other conditions, including persons, settings, treatment variables, and
measurement variables. Construct validity refers to how concepts are operational-
ized as experimental measures. Conclusion validity refers to inferences about the
relationship between treatment and outcome variables.
The accomplishment of empirical studies relies on performing well-defined and
evolutionary activities. The classical empirical study process consists of five phases:
definition, planning, operation, analysis, and interpretation, as well as reporting and
packaging (Juristo and Moreno 2001; Malhotra 2016). The definition phase makes
the investigated problem and overall objectives of the study explicit. The planning
phase covers the study design and includes the definition of research questions and
hypotheses as well as the definition of data collection, data analysis, and validity
procedures. In the operation phase, the study is actually conducted. In the analysis
and interpretation phase, the collected data is analyzed, assessed, and discussed.
Finally, in the reporting and packaging phase, the results of the study are reported
(e.g., in a journal article, a conference paper, or a technical report) and suitably
packaged to provide study material and data. The latter has become more critical
recently due to the open science movement (see chapter “Open Science in Software
Engineering”).

3 Evolution of Empirical Software Engineering

The application of empirical methods in general and empirical software engineering

in particular is well-established in software engineering research. Almost all papers
published in major software engineering venues these days include an empirical
study (Theisen et al. 2017). Furthermore, since 2000, research methodology has
received considerable attention in the software engineering research community
resulting in many available publications on empirical research methodology in
software engineering. In a recent mapping study, Molléri et al. (2019) identified
341 methodological papers on empirical research in software engineering.
The application of empirical methods and the underlying research methodology
has developed iteratively since the foundation of software engineering in the 1960s.
Guéhéneuc and Khomh (2019) discuss landmark articles, books, and venues in
empirical software engineering that indicate the iterative development of the field.
Bird et al. (2015) distinguish four “generations” of analyzing software data, i.e.,
8 M. Felderer and G. H. Travassos

preliminary work, academic experiments, industrial experiments, and “data science

everywhere.” In this section, we present five iterations of the development of
empirical software engineering from a methodological perspective. We additionally
take articles and venues into account, which is needed for a holistic understanding
of the field’s development. We complement our presentation of the evolution of
empirical software engineering by presenting the current situation and an outlook in
Sect. 4 and the available books on empirical software engineering in chronological
order in Sect. 5 on recommended further reading.

3.1 First Iteration: Mid-1960s to Mid-1970s

In the early years of software engineering, empirical studies were rare, and the
only research model commonly in use was the analytical method, where different
formal theories were advocated devoid of any empirical evaluation (Glass 1994).
According to a systematic literature review of empirical studies performed by
Zendler (2001), Grant and Sackman (1967) published the first empirical study in
software engineering in 1967. The authors conducted an experiment that compared
the performance of two groups of developers, one working with online access to
a computer through a terminal and the other with offline access in batch mode.
Another empirical study published early in the history of software engineering was
an article by Knuth (1971), in which the author studied a set of Fortran programs
to understand what developers do in Fortran programs. Akiyama (1971) describes
the first known “size law” (Bird et al. 2015), stating that the number of defects is a
function of the number of lines of code. The authors in these and other early studies
defined the goal of the study, the questions to research, and the measures to answer
these questions in an ad hoc fashion (Guéhéneuc and Khomh 2019). However, they
were pioneers in the application of empirical methods in software engineering.

3.2 Second Iteration: Mid-1970s to Mid-1980s

In the second iteration, already more empirical studies, mainly in vitro experiments,
were conducted. Prominent examples are experiments on structured program-
ming (Lucas et al. 1976), flowcharting (Shneiderman et al. 1977), and software
testing (Myers 1978). The second iteration is characterized by first attempts to pro-
vide a systematic methodology to define empirical studies in software engineering in
general and experiments in particular. These attempts culminated in the definition of
the Goal/Question/Metrics (GQM) approach by Basili and Weiss (1984). The GQM
approach helped practitioners and researchers to define measurement programs
based on goals related to products, processes, and resources that can be achieved
by answering questions that characterize the objects of measurement using metrics.
The Evolution of Empirical Methods in Software Engineering 9

The methodology has been used to define experiments in software engineering

systematically.
In that iteration, empirical software engineering was also institutionalized for
the first time. In 1976, the NASA Goddard Software Engineering Laboratory
(NASA/SEL) was established at the University of Maryland, College Park (USA),
aiming to support the observation and understanding of software projects (Basili
and Zelkowitz 2007). The establishment of NASA/SEL provided the means to
strengthen the importance of using basic scientific and engineering concepts in
the context of software engineering (McGarry et al. 1994). The paradigm change
provided by using GQM (Basili and Weiss 1984), including the ability of packaging
knowledge on how to better build a software system, improved the way experiences
could be organized and shared. The building and evolution of models at NASA/SEL
pave the road for organizing the Experience Factory model (Basili et al. 1994) and
the dissemination of initial good practices on empirical software engineering.

3.3 Third Iteration: Mid-1980s to End of the 1990s

In the third iteration, not only experiments but also surveys (for instance,
by Burkhard and Jenster (1989) on the application of computer-aided software
engineering tools) and case studies (for instance, by Curtis et al. (1988) on the
software design process for large systems) were performed to some extent. Also,
the explicit discussion of threats to validity appeared in that iteration. One of the first
studies explicitly discussing its threats to validity was an article by Swanson and
Beath (1988) on the use of case study data in software management research. From
the late 1980s, researchers also started to analyze software data using algorithms
taken from artificial intelligence research (Bird et al. 2015). For instance, decision
trees and neural networks were applied to predict error-proneness (Porter and
Selby 1990), to estimate software effort (Srinivasan and Fisher 1995) and to model
reliability growth (Tian 1995).
In the third iteration, empirical studies began to attract the attention of several
research groups all over the world, who realized the importance of providing empir-
ical evidence about the developed and investigated software products and processes.
The experiences shared by NASA/SEL and the participation of several researchers
in conducting experiments together with NASA/SEL helped to strengthen the use
of different experimental strategies and the application of surveys.
The interest in the application of the scientific method by different researchers,
the identification of the need to evolve the experimentation process through sharing
of experimental knowledge among peers as well as the transfer of knowledge to
industry, among other reasons, led to the establishment of the International Software
Engineering Research Network (ISERN) in 1992. ISERN held its first annual
meeting in Japan in 1993 sponsored by the Graduate School of Information Science
at the Nara Institute of Science and Technology.
10 M. Felderer and G. H. Travassos

The need to share the ever increasing number of studies and their results and the
growing number of researchers applying empirical methods in software engineering
lead to the foundation of suitable forums. In 1993 the IEEE International Software
Metrics Symposium, in 1996, the Empirical Software Engineering International
Journal, and in 1997, the Empirical Assessments in Software Engineering (EASE)
event at Keele University were founded.
By the end of this iteration, several institutes dedicated to empirical software
engineering were established. In 1996, the Fraunhofer Institute for Experimental
Software Engineering (IESE) associated with the University of Kaiserslautern
(Germany) was established. In 1998, the Fraunhofer Center for Experimental
Software Engineering (CESE) associated with the University of Maryland, College
Park (USA) began operations. Also, other institutions and laboratories, such as
National ICT Australia as well as the Simula Research Laboratory and SINTEF
(both located in Norway), among others, started to promote empirical studies in
software engineering in the industry.
Finally, by the end of the 1990s, the publication of methodological papers on
empirical methods in software engineering started. Zelkowitz and Wallace (1998)
provided an overview of experimental techniques to validate new technologies,
Seaman (1999) provided guidelines for qualitative data collection and analysis, and
Basili et al. (1999) discussed families of experiments.

3.4 Fourth Iteration: The 2000s

Since 2000 research methodology has received considerable attention, and therefore
the publication of methodological papers further increased. For instance, Höst et al.
(2000) discuss the usage of students as subjects in experiments, Shull et al. (2001)
describe a methodology to introduce software processes based on experimentation,
Pfleeger and Kitchenham (2001) provide guidelines on surveys in software engi-
neering, Lethbridge et al. (2005) provide a classification of data collection methods,
Kitchenham and Charters (2007) provide guidelines for performing systematic
literature reviews in software engineering, Shull et al. (2008) discuss the role
of replication in empirical software engineering, and Runeson and Höst (2009)
provide guidelines for case study research. In connection to the increased interest
in research methodology, also the first books on empirical research methods in
software engineering with a focus on experimentation written by Wohlin et al.
(2000) and Juristo and Moreno (2001) appeared around 2000 (see Sect. 5 for
a comprehensive overview of books on empirical software engineering). Also,
combining research methods and performing multi-method research became more
popular in the period. One of the first papers following a multi-method research
methodology was published by Espinosa et al. (2002) on shared mental models,
familiarity, and coordination in distributed software teams.
With the growing number of empirical studies, knowledge aggregation based
on these primary studies became more crucial to understand software engineering
The Evolution of Empirical Methods in Software Engineering 11

phenomena better. No single empirical study on a software engineering phe-

nomenon can be considered definitive (Shull et al. 2004) and generalized to any
context. Therefore, the replication of studies in different contexts is of paramount
importance to strengthen its findings. However, the existence of conclusive, no
conclusive, contradictory, and confirmatory results about a particular software
engineering phenomenon should be combined to strengthen the evidence on the
software phenomena or to reveal the need for more primary studies on phenomenon
of interest. In consequence, there arose a need for secondary studies that aim
to organize, aggregate, and synthesize all relevant results from primary studies
regarding a particular phenomenon under research. Kitchenham (2004) was the
first who recommended the use of systematic literature reviews (SLRs) in software
engineering and adapted respective guidelines, mainly from medical research, to
software engineering. With the guidelines of Kitchenham (2004) and Biolchini
(2005), the empirical software engineering community had a tool to systemat-
ically synthesize knowledge available in primary studies, which spread rapidly
and enabled evidence-based software engineering (Kitchenham et al. 2004). In
a systematic review of SLRs in software engineering, Kitchenham and Brereton
(2013) identified 68 papers reporting 63 unique SLRs published in SE conferences
and journals between 2005 and mid-2012. Petersen et al. (2008) clarify and expand
upon the differences between SLRs and systematic mapping studies and provide
guidelines for the latter. In their seminal paper on the future of empirical methods
in software engineering research, Sjøberg et al. (2007) present the important role
of synthesis of empirical evidence in their vision of software engineering research.
The vision is that for all fields of software engineering, empirical research methods
should enable the development of scientific knowledge about how useful different
SE technologies are for different kinds of actors, performing different kinds of
activities, on different kinds of systems to guide the development of new SE
technology and important SE decisions in industry. Major challenges to the pursuit
of this vision are more and better synthesis of empirical evidence, and connected
to that building and testing more theories as well as increasing quality, including
relevance, of studies.
One of the problems faced by the software engineering community has often
been the scarcity of software data for conducting empirical studies (Malhotra 2016).
The availability of (open) source code repositories and software process data due
to automated or even continuous software engineering enabled new data mining
approaches in software engineering in that period. In a seminal paper, Zimmermann
et al. (2005) used association rule learning to find patterns of defects in a large
set of open-source projects. Furthermore, also, software data from companies were
analyzed. For instance, at AT&T, Ostrand et al. (2004) used code metrics to
predict defects, and at Microsoft—which even founded an own Empirical Software
Engineering (ESE) group in Microsoft Research (Bird et al. 2011)—Nagappan and
Ball (2005) showed that data from that organization could predict software quality.
In consequence, also repositories—like the PROMISE repository—that collect
software data and make them publicly available were founded. The PROMISE
repository was founded in 2005 and seeded with NASA data (Menzies et al. 2014).
12 M. Felderer and G. H. Travassos

The empirical evidence gathered through analyzing the data collected from the
software repositories is considered to be an important support for the (empirical)
software engineering community these days. There are even venues that focus on
analysis of software data such as Mining Software Repositories (MSR), which was
organized for the first time in 2004 in Edinburgh (UK) and Predictive Models and
Data Analytics in Software Engineering (PROMISE), which was organized for the
first time in 2005 in St. Louis (USA).
In general, the growing interest in empirical software engineering in that period
resulted in projects such as the Experimental Software Engineering Research
Network (ESERNET) in Europe from 2001 to 2003 and the foundation of several
venues. In 2007, the first ACM/IEEE International Symposium on Empirical
Software Engineering and Measurement (ESEM) was held in Madrid (Spain).
ESEM is the result of the merger between the ACM/IEEE International Symposium
on Empirical Software Engineering, which ran from 2002 to 2006, and the IEEE
International Software Metrics Symposium, which ran from 1993 to 2005. In 2003,
Experimental Software Engineering Latin American Workshop (ESELAW) was
organized for the first time. Also, in 2003, the International Advanced School
on Empirical Software Engineering (IASESE) performed its first set of classes in
Rome (Italy). In 2006, the International Doctoral Symposium on Empirical Software
Engineering (IDoESE) was founded. Today, the ISERN annual meeting, IASESE,
IDoESE, and ESEM form the Empirical Software Engineering International Week
(ESEIW), which is held annually.

3.5 Fifth Iteration: The 2010s

Since 2010 empirical studies are “everywhere” in software engineering. Almost

all papers in major software engineering conferences like ICSE contain empirical
studies. Also, more and more books dedicated to empirical research methodology
in software engineering are published (see Sect. 5), and papers on empirical research
methodology are published at a constant pace. For instance, Ivarsson and Gorschek
(2011) present a model for evaluating the rigor and relevance of technology
evaluations in industry, Arcuri and Briand (2014) provide a guide to statistical tests
for assessing randomized algorithms in software engineering, Wieringa (2014a)
discusses scaling up of empirical methods for technology validation in practice,
Wohlin and Aurum (2015) provide a decision-making structure for selecting a
research design, de Mello et al. (2015) provide probabilistic sampling approaches
for large-scale surveys, Sharp et al. (2016) discuss the use and value of ethnographic
studies in software engineering research, Stol et al. (2016) discuss the use of
grounded theory and their reporting, Briand et al. (2017) discuss the importance
of context and the overrating of generalizability in software engineering, and
Stol and Fitzgerald (2018) provide a holistic framework for software engineering
research. Furthermore, Harman et al. (2010) provide a comprehensive overview and
guidance on the application of search-based optimization in software engineering.
The Evolution of Empirical Methods in Software Engineering 13

Especially, in this period many papers presenting results on search-based software

engineering, that generally (though not exclusively) fall in the category of empirical
software engineering papers were published. Due to the potentially high compu-
tational complexity of optimization algorithms, some researchers have started to
use high performance computing environments to support the execution of their
studies (Farzat et al. 2019).
In this iteration, one can observe a growing interest in the role of theory within
software engineering research to develop the field further as a scientific discipline. In
December 2009, the Software Engineering Method and Theory (SEMAT) initiative
was launched that aims towards the development of a general theory of software
engineering. SEMAT organized several events, among others, a workshop series
on a General Theory of Software Engineering (GTSE) between 2012 and 2015.
Stol and Fitzgerald (2015) even argue for a theory-oriented software engineering
research perspective, which can complement the recent focus on evidence-based
software engineering. Also, several concrete theories have been developed in that
iteration. For instance, Johnson and Ekstedt (2016) present a general theory of
software engineering called Tarpit, Bjarnason et al. (2016) a theory of distances
in software engineering, and Wagner et al. (2019) a theory on requirements
engineering.
Today not only almost all papers in major software engineering conferences
contain empirical studies, but also most software engineering conferences have
explicitly integrated empirical software engineering into their program, e.g., as
dedicated sessions or tracks. In addition, there are several workshops on conducting
empirical studies in specific areas. For instance, at ICSE, there has been a collocated
International Workshop on Conducting Empirical Studies in Industry (CESI) and at
RE the International Workshop on Empirical Requirements Engineering (EmpiRE).
The Experimental Software Engineering Latin American Workshop (ESELAW)
joined the Ibero-American Conference on Software Engineering (CIbSE) in 2011
as a colocated workshop and became a dedicated track in 2013 due to the increased
number of submissions.
ESEIW, including ESEM, and EASE are established as the two leading annual
events to discuss methodological issues on empirical research in software engi-
neering. Empirical methods have been an explicit topic in several summer schools
including the annual LASER summer school (which hosted the topic empirical
software engineering in 2010), PASED—Canadian Summer School on Practical
Analyses of Software Engineering Data in 2011, the Empirical Research Meth-
ods in Software Engineering and Informatics (ERMSEI) in 2016 and 2017, the
International Summer School on Software Engineering (SIESTA) in 2018 and 2019
as well as the 2019 Summer School in Empirical Software Engineering at Brunel
(UK). In the context of ESEIW, the International Advanced School on Empirical
Software Engineering (IASESE) has been organized annually since 2003 and helped
to spread knowledge on current empirical methods in software engineering among
junior and senior researchers. Figure 1 presents the IASESE timeline and its topics
along the places and years. The topics taught over the years also reflect the evolution
of empirical software engineering, as discussed in this section.
 14

2003 – Rome, Italy 2019 – Porto de Galinhas, Brazil

Evaluating the maturing of a technology for use Observation as a Data Collection Technique
Using Empirical study to do technology transfer for Software Engineering Research
Using qualitative analysis in software engineering
Building parametric models 2018 – Oulo, Finland
Empirical Software Engineering and Data
2004 – Los Angeles, USA
Science: Old Wine in a New Bottle
How to do Empirical Research IASESE
2005 – Noosa Heads, Australia International Advanced School on 2017 – Toronto, Canada
Research Protocols and Systematic Empirical Software Engineering Product Development and Innovation with
Literature Reviews Continuous Experimentation
2006 – Rio de Janeiro, Brazil 2016 – Ciudad Real, Spain
Technology Evaluation Locations and Topics
Surveys in Software Engineering
How to run empirical studies using project 2003 - 2019
repositories (and avoid common pitfalls) 2015 – Beijing, China
The role of replication in Software Engineering Theories in Empirical Software Engineering
Software engineering process simulation
2007 – Madrid, Spain 2014 – Torino, Italy
Case Study Research In Vivo Experimentation in Software
Engineering
2008 – Kaiserslautern, Germany
Replication and Aggregation of Software 2013 – Baltimore, USA
Engineering Experiments Action Research
2009 – Orlando, USA
Selecting Research Methods for Empirical 2012 – Lund, Sweden
Software Engineering Evidence Synthesis of Qualitative Research

2010 – Bolzano-Bozen, Italy 2011 – Banff, Canada

Using Ethnographic Methods in Empirical Evidence-based Decision-Support in
Software Engineering Research Software Engineering

Fig. 1 International Advanced School on Empirical Software Engineering (IASESE) timeline and topics from 2003 to 2019
M. Felderer and G. H. Travassos
The Evolution of Empirical Methods in Software Engineering 15

4 Current Situation and Outlook

Since the first empirical studies in the 1960s, the field of empirical software
engineering has considerably matured in several iterations. However, the empirical
methods resulting from the five iterations presented in the previous section are
not the end of the story, and as in any scientific discipline, research methods
develop further. The chapters of this book discuss contemporary empirical methods
that impact the current evolution of empirical software engineering and form the
backbone of its next iteration. For sure, the description of the current situation
and future trends is never complete and always subjective to some extent. But
we think that the chapters covered in this book show several interesting trends
in contemporary empirical methods in software engineering, which we want to
summarize here.
The evolution of empirical software engineering leads to the continuous adoption
of empirical methods from other fields and the refinement of existing empirical
methods in software engineering. The resulting plurality of research methods
requires guidance in knowledge-seeking and solution-seeking (i.e., design science)
research. The chapter “Guidelines for Conducting Software Engineering Research”
presents guidelines for conducting software engineering research based on the
ABC framework, where ABC represents the three desirable aspects of research
generalizability over actors (A), precise control of behavior (B), and realism of
context (C). Each empirical method has its strengths and weaknesses. It is beneficial
to utilize a mix of methods depending on the research goal or even to combine
methods. Case survey research combines case study and survey research, which rely
primarily on qualitative and quantitative data, respectively. The chapter “Guidelines
for Case Survey Research in Software Engineering” provides an overview of the
case survey method. While being an important and often used empirical method,
survey research has been less discussed on a methodological level than other types
of empirical methods. The chapter “Challenges in Survey Research” discusses
methodological issues in survey research for software engineering concerning
theory building, sampling, invitation and follow-up, statistical analysis, qualitative
analysis, and assessment of psychological constructs. Although software engineer-
ing is an engineering discipline, the design science paradigm has been explicitly
adapted to software engineering relatively late by Wieringa (2014b), and the full
potential of the design science paradigm has not been exploited so far in software
engineering. The chapter “The Design Science Paradigm as a Frame for Empirical
Software Engineering” uses the design science paradigm as a frame for empirical
software engineering and uses it to support the assessment of research contributions,
industry-academia communication, and theoretical knowledge building.
It is generally acknowledged that software development is a human-intensive
activity as software is built by humans for humans. However, traditionally SE
research has focused on artifacts and processes without explicitly taking human
factors in general and the developer perspective in particular into account. If the
perspective on how developers work was considered, then it was mostly measured
16 M. Felderer and G. H. Travassos

from a subjective perspective, e.g., by interviews or opinion surveys, or a “black

box” perspective by mining repository data or measuring the created development
artifacts. The chapter “Biometric Measurement in Software Engineering” introduces
biometric sensors and measure that provide new opportunities to more objectively
measure physiological changes in the human body that can be linked to various
psychological processes. These biometric measurements can be used to gain insights
on fundamental cognitive and emotional processes of software developers while
they are working, but also to provide better and more prompt tool support for
developers. Another human-related issue is the involvement of humans in empirical
studies, especially in experiments. On the one hand, it is normally difficult to recruit
a significant number of professionals for an empirical study, and on the other
hand, measurements are invasive. The chapter “Empirical Software Engineering
Experimentation with Human Computation” explores the potential of human com-
putation methods, such as crowdsourcing, for experimentation in empirical software
engineering.
Empirical methods rely on the collected data. However, the volume, velocity,
and variety of data in software products and processes have exploded during the last
years. Therefore, the new scientific paradigm of data science has gained much atten-
tion, also within software engineering. The chapter “Data Science and Empirical
Software Engineering” relates to traditional ESE and data science. It shows that both
paradigms have many characteristics in common and can benefit from each other.
Given large data sets, optimization is an important form of data analytics support of
human decision-making. Empirical studies serve both as a model and as data input
for optimization. The chapter “Optimization in Software Engineering: A Pragmatic
Approach” provides an overview of optimization in software engineering in general
and its value and applicability in ESE in particular. With increased automation,
uncertainty (due to the application of statistical models), and monitoring capabilities
in data-driven software engineering, also the role of simulation techniques becomes
more important. The chapter “The Role of Simulation-Based Studies in Software
Engineering Research” provides a guide to simulation-based studies in software
engineering. Bayesian data analysis is a means to embrace uncertainty by using
multilevel statistical models and making use of all available information at hand.
The chapter “Bayesian Data Analysis in Empirical Software Engineering: The
Case of Missing Data” provides an introduction to Bayesian data analysis and an
application example to empirical software engineering dealing with common issues
in ESE like missing data.
Extracting, aggregating, and synthesizing evidence from empirical studies is
essential for the development of scientific knowledge and the field of software engi-
neering. However, conducting secondary studies like systematic literature reviews
and aggregating evidence is still challenging. Conducting systematic literature
reviews (SLRs) is largely a manual and, therefore, time-consuming and error-
prone process. The chapter “Automating Systematic Literature Review” provides
strategies to automate the SLR process. Secondary studies often lack connection
to software engineering practice, which is essential to software engineering. The
chapter “Rapid Reviews in Software Engineering” presents the concept of rapid
The Evolution of Empirical Methods in Software Engineering 17

reviews, which are lightweight secondary studies focused on delivering evidence to

practitioners on time. Another approach to link to practice is to take grey literature
into account in empirical studies. The chapter “Benefitting from the Grey Literature
in Software Engineering Research” discusses the concept of grey literature in
software engineering and ways how to consider it in primary and secondary studies.
Considering that secondary studies are often used to support the evidence-based
paradigm, it is crucial to managing their threats properly. The chapter “Guidelines
for Managing Threats to Validity of Secondary Studies in Software Engineering”
provides guidelines for managing threats to validity of secondary studies in software
engineering. Evidence in software engineering is often rare and produced in both
quantitative and qualitative forms. It makes the synthesis of evidence, which is
an essential element in scientific knowledge creation, a challenging task. The
chapter “Research Synthesis in Software Engineering” provides an overview of
research synthesis methods in software engineering.
Society in general and funding agencies in particular put a stronger focus on
the impact of (software engineering) research. Therefore, open science and research
transfer are becoming essential topics in (empirical) software engineering. Open
science describes the movement of making any research artifact available to the
public and includes open access, open data, and open source. The topic is natural and
especially important in empirical software engineering to guarantee the replicability
of empirical studies. The chapter “Open Science in Software Engineering” reflects
upon the essentials in open science for software engineering to help to establish
a common ground and to make open science a norm in SE. Industry-academia
collaboration is one of the cornerstones of empirical software engineering. However,
close and sustainable collaboration with industry are key issues in the field.
The chapter “Third Generation Industrial Co-production in Software Engineering”
presents a seven-step industrial coproduction approach that enables deep and long-
term industry-academia collaboration.

5 Recommended Further Reading

Since 2000 research methodology has received considerable attention in the soft-
ware engineering research community. Therefore, plenty of literature is available
on empirical research methodology in software engineering. Molléri et al. (2019)
identified in a recent systematic mapping study 341 methodological papers on
empirical research in software engineering—and therefore, a complete overview
would exceed the scope of this book chapter. However, following the style of this
book chapter, we provide an overview of the available English text and special
issue books explicitly dedicated to empirical research methodology in software
engineering in chronological order of their publication.
Wohlin et al. (2000) published a book entitled “Experimentation in Software
Engineering,” which provides an overview of the core empirical strategies in
software engineering, i.e., surveys, experimentation, and case studies and as its
18 M. Felderer and G. H. Travassos

main content all steps in the experimentation process, i.e., scoping, planning,
operation, analysis and interpretation as well as presentation and package. The
book is complemented by exercises and examples, e.g., an experiment comparing
different programming languages. Consequently, the book targets students, teachers,
researchers, and practitioners in software engineering. In 2012 a revision of this
popular book had been published with Springer (Wohlin et al. 2012).
Juristo and Moreno (2001) published a book entitled “Basics of Software Engi-
neering Experimentation,” which presents the basics of designing and analyzing
experiments both to software engineering researchers and practitioners based on
SE examples like comparing the effectiveness of defect detection techniques. The
book presents the underlying statistical methods, including the computation of test
statistics in detail.
Endres and Rombach (2003) published “A Handbook of Software and Systems
Engineering. Empirical Observations, Laws, and Theories.” The book presents rules,
laws, and their underlying theories from all phases of the software development
lifecycle. The book provides the reader with clear statements of software and system
engineering laws and their applicability as well as related empirical evidence. The
consideration of empirical evidence distinguishes the book from other available
handbooks and textbooks on software engineering.
Juristo and Moreno (2003) edited “Lecture Notes on Empirical Software Engi-
neering,” which aims to spread the idea of the importance of empirical knowledge
in software development from a highly practical viewpoint. It defines the body of
empirically validated knowledge in software development to advise practitioners on
what methods or techniques have been empirically analyzed and what the results
were. Furthermore, it promotes “empirical tests,” which have traditionally been
carried out by universities or research centers, for application in industry to validate
software development technologies used in practice.
Shull et al. (2007) published the “Guide to Advanced Empirical Software Engi-
neering.” It is an edited book written by experts in empirical software engineering. It
covers advanced research methods and techniques, practical foundations, as well as
knowledge creation, approaches. The book at hand provides a continuation of that
seminal book covering recent developments in empirical software engineering.
Runeson et al. (2012) published a book entitled “Case Study Research in
Software Engineering: Guidelines and Examples,” which covers guidelines for
all steps of case study research, i.e., design, data collection, data analysis and
interpretation, as well as reporting and dissemination. The book is complemented
with examples from extreme programming, project management, quality monitoring
as well as requirements engineering and additionally also provides checklists.
Wieringa (2014b) published a book entitled “Design Science Methodology for
Information Systems and Software Engineering,” which provides guidelines for
practicing design science in software engineering research. A design process usually
iterates over two activities, i.e., first designing an artifact that improves something
for stakeholders, and subsequently empirically validating the performance of that
artifact in its context. This “validation in context” is a key feature of the book.
The Evolution of Empirical Methods in Software Engineering 19

Menzies et al. (2014) published a book entitled “Sharing Data and Models in
Software Engineering.” The central theme of the book is how to share what has been
learned by data science from software projects. The book is driven by the PROMISE
(Predictive Models and Data Analytics in Software Engineering) community. It is
the first book dedicated to data science in software and mining software repositories.
Closely related to this book, Bird et al. (2015) published a book entitled “The Art
and Science of Analyzing Software Data,” which is driven by the MSR (Mining
Software Repositories) community and focuses mainly on data analysis based on
statistics and machine learning. Another related book published by Menzies et al.
(2016) covers perspectives on data science for software engineering by various
authors.
Kitchenham et al. (2015) published a book entitled “Evidence-based Software
Engineering and Systematic Reviews,” which provides practical guidance on how
to conduct secondary studies in software engineering. The book also discusses the
nature of evidence and explains the types of primary studies that provide inputs to a
secondary study.
Malhotra (2016) published a book entitled “Empirical Research in Software
Engineering: Concepts, Analysis, and Applications,” which shows how to imple-
ment empirical research processes, procedures, and practices in software engineer-
ing. The book covers many accompanying exercises and examples. The author
especially also discusses the process of developing predictive models, such as defect
prediction and change prediction, on data collected from source code repositories,
and, more generally the application of machine learning techniques in empirical
software engineering.
ben Othmane et al. (2017) published a book entitled “Empirical Research
for Software Security: Foundations and Experience,” which discusses empirical
methods with a special focus on software security.
Staron (2019) published a book entitled “Action Research in Software Engineer-
ing: Theory and Applications,” which offers a comprehensive discussion on the use
of action research as an instrument to evolve software technologies and promote
synergy between researchers and practitioners.
In addition to these textbooks, there are also edited books available that are
related to special events in empirical software engineering and cover valuable
methodological contributions.
Rombach et al. (1993) edited proceedings from a Dagstuhl seminar in 1992
on empirical software engineering entitled “Experimental Software Engineering
Issues: Critical Assessment and Future Directions.” The goal was to discuss the state
of the art of empirical software engineering by assessing past accomplishments,
raising open questions, and proposing a future research agenda at that time.
However, many contributions of that book are still relevant today.
Conradi and Wang (2003) edited a book entitled “Empirical Methods and Studies
in Software Engineering: Experiences from ESERNET,” which covers experiences
from the Experimental Software Engineering Research NETwork (ESERNET), a
thematic network funded by the European Union between 2001 and 2003.
20 M. Felderer and G. H. Travassos

Boehm et al. (2005) edited a book entitled “Foundations of Empirical Software

Engineering: The Legacy of Victor R. Basili” on the occasion of V. R. Basili’s 65th
birthday, which covers reprints of 20 papers that defined much of his work.
Basili et al. (2007) edited proceedings from another Dagstuhl seminar in 2006
on empirical software engineering entitled “Empirical Software Engineering Issues.
Critical Assessment and Future Directions.”
Münch and Schmid (2013) edited a book entitled “Perspectives on the Future
of Software Engineering: Essays in Honor of Dieter Rombach” on the occasion
of Dieter Rombach’s 60th birthday, which covers contributions by renowned
researchers and colleagues of him.

6 Conclusion

In this chapter we presented the evolution of empirical software engineering in

five iterations, i.e., (1) mid-1960s to mid-1970s, (2) mid-1970s to mid-1980s, (3)
mid-1980s to end of the 1990s, (4) the 2000s, and (5) the 2010s. We presented
the five iterations of the development of empirical software engineering mainly
from a methodological perspective and additionally took key papers, venues, and
books into account. Available books explicitly dedicated to empirical research
methodology in software engineering were covered in chronological order in a
separate section on recommended further readings. Furthermore, we discuss—
based on the chapters in this book—trends on contemporary empirical methods in
software engineering related to the plurality of research methods, human factors,
data collection and processing, aggregation and synthesis of evidence, and impact
of software engineering research.

Acknowledgements We thank all the authors and reviewers of this book on contemporary
empirical methods in software engineering for their valuable contribution.

References

Akiyama F (1971) An example of software system debugging. In: IFIP congress (1), vol 71. North-
Holland, Amsterdam, pp 353–359
Araújo MAP, Monteiro VF, Travassos GH (2012) Towards a model to support studies of software
evolution. In: Proceedings of the ACM-IEEE international symposium on empirical software
engineering and measurement (ESEM ’12). ACM, New York, pp 281–290
Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized
algorithms in software engineering. Softw Test Verification Reliab 24(3):219–250
Baros MO, Werner CML, Travassos GH (2004) Supporting risks in software project management.
J Syst Softw 70(1):21–35
Basili VR (1993) The experimental paradigm in software engineering. In: Experimental software
engineering issues: critical assessment and future directions. Springer, Berlin, pp 1–12
The Evolution of Empirical Methods in Software Engineering 21

Basili VR, Weiss DM (1984) A methodology for collecting valid software engineering data. IEEE
Trans Softw Eng SE-10(6):728–738
Basili VR, Zelkowitz MV (2007) Empirical studies to build a science of computer science.
Commun Assoc Comput Mach 50(11):33–37
Basili VR, Caldiera G, Rombach HD (1994) Experience factory. Encycl Softw Eng 1:469–476
Basili VR, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE
Trans Softw Eng 25(4):456–473
Basili V, Rombach D, Schneider K, Kitchenham B, Pfahl D, Selby R (2007) Empirical software
engineering issues. In: Critical assessment and future directions: international workshop,
Dagstuhl Castle, June 26–30, 2006, Revised Papers, vol 4336. Springer, Berlin
ben Othmane L, Jaatun MG, Weippl E (2017) Empirical research for software security: foundations
and experience. CRC Press, Boca Raton
Bertin E (1978) Qualitative and semiquantitative analysis. Springer, Berlin, pp 435–457
Biolchini MPNATG J (2005) Systematic review in software engineering: relevance and utility.
Technical report
Bird C, Murphy B, Nagappan N, Zimmermann T (2011) Empirical software engineering at
Microsoft research. In: Proceedings of the ACM 2011 conference on computer supported
cooperative work. ACM, New York, pp 143–150
Bird C, Menzies T, Zimmermann T (2015) The art and science of analyzing software data. Elsevier,
Amsterdam
Bjarnason E, Smolander K, Engström E, Runeson P (2016) A theory of distances in software
engineering. Inf Softw Technol 70:204–219
Boehm B, Rombach HD, Zelkowitz MV (2005) Foundations of empirical software engineering:
the legacy of Victor R. Basili. Springer, Berlin
Briand L, Bianculli D, Nejati S, Pastore F, Sabetzadeh M (2017) The case for context-driven
software engineering research: generalizability is overrated. IEEE Softw 34(5):72–75
Burkhard DL, Jenster PV (1989) Applications of computer-aided software engineering tools:
survey of current and prospective users. ACM SIGMIS Database Database Adv Inf Syst
20(3):28–37
Conradi R, Wang AI (2003) Empirical methods and studies in software engineering: experiences
from ESERNET, vol 2765. Springer, Berlin
Creswell JW, Creswell JD (2018) Research design: qualitative, quantitative, and mixed methods
approaches. SAGE, Los Angeles
Curtis B, Krasner H, Iscoe N (1988) A field study of the software design process for large systems.
Commun Assoc Comput Mach 31(11):1268–1287
de Mello RM, Da Silva PC, Travassos GH (2015) Investigating probabilistic sampling approaches
for large-scale surveys in software engineering. J Softw Eng Res Dev 3(1):8
Endres A, Rombach HD (2003) A handbook of software and systems engineering: empirical
observations, laws, and theories. Pearson Education, Old Tappan
Espinosa A, Kraut R, Slaughter S, Lerch J, Herbsleb J, Mockus A (2002) Shared mental
models, familiarity, and coordination: a multi-method study of distributed software teams. In:
Proceedings of ICIS 2002, p 39
Farzat F, Barros MO, Travassos GH (2019) Evolving JavaScript code to reduce load time. IEEE
Trans Softw Eng
Fink A (2003) The survey handbook. SAGE, Los Angeles
Glass RL (1994) The software-research crisis. IEEE Softw 11(6):42–47
Grant EE, Sackman H (1967) An exploratory investigation of programmer performance under on-
line and off-line conditions. IEEE Trans Hum Factors Electron 1:33–48
Guéhéneuc YG, Khomh F (2019) Empirical software engineering. In: Cha S, Taylor RN, Kang KC
(eds) Handbook of software engineering. Springer, Berlin, pp 285–320
Harman M, McMinn P, De Souza JT, Yoo S (2010) Search based software engineering: techniques,
taxonomy, tutorial. In: Empirical software engineering and verification. Springer, Berlin,
pp 1–59
Harrison W, Basili VR (1996) Editorial. Empir Softw Eng 1:5–10
22 M. Felderer and G. H. Travassos

Hey AJ, Hey T, Pápay G (2014) The computing universe: a journey through a revolution.
Cambridge University Press, Cambridge
Höst M, Regnell B, Wohlin C (2000) Using students as subjects—a comparative study of students
and professionals in lead-time impact assessment. Empir Softw Eng 5(3):201–214
IEEE (1990) 610.12-19919—IEEE standard glossary of software engineering terminology. IEEE,
New York
IEEE (2010) ISO/IEC/IEEE 24765:2010 systems and software engineering—vocabulary. IEEE,
Geneva
Ivarsson M, Gorschek T (2011) A method for evaluating rigor and industrial relevance of
technology evaluations. Empir Softw Eng 16(3):365–395
Johnson P, Ekstedt M (2016) The Tarpit–a general theory of software engineering. Inf Softw
Technol 70:181–203
Juristo N, Moreno AM (2001) Basics of software engineering experimentation. Springer, Berlin
Juristo N, Moreno AM (2003) Lecture notes on empirical software engineering, vol 12. World
Scientific, New Jersey
Kitchenham B (2004) Procedures for performing systematic reviews. Technical report
Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in
software engineering. Inf Softw Technol 55(12):2049–2075
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in
software engineering. Technical report
Kitchenham BA, Dybå T, Jorgensen M (2004) Evidence-based software engineering. In: Proceed-
ings of the 26th international conference on software engineering. IEEE Computer Society,
Silver Spring, pp 273–281
Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and system-
atic reviews, vol 4. CRC Press, Boca Raton
Knuth DE (1971) An empirical study of Fortran programs. Softw Pract Exp 1(2):105–133
Lethbridge TC, Sim SE, Singer J (2005) Studying software engineers: data collection techniques
for software field studies. Empir Softw Eng 10(3):311–341
Lucas J, Henry C, Kaplan RB (1976) A structured programming experiment. Comput J 19(2):136–
138
Lyu MR, et al (1996) Handbook of software reliability engineering, vol 222. IEEE Computer
Society Press, Los Alamitos
Malhotra R (2016) Empirical research in software engineering: concepts, analysis, and applica-
tions. Chapman and Hall/CRC, London
McGarry F, Pajerski R, Page G, Waligora S, Basili V, Zelkowitz M (1994) Software process
improvement in the NASA software engineering laboratory. Technical report, CMU/SEI-
94-TR-022. Software Engineering Institute/Carnegie Mellon University, Pittsburgh. http://
resources.sei.cmu.edu/library/asset-view.cfm?AssetID=12241
Menzies T, Kocaguneli E, Turhan B, Minku L, Peters F (2014) Sharing data and models in software
engineering. Morgan Kaufmann, Amsterdam
Menzies T, Williams L, Zimmermann T (2016) Perspectives on data science for software
engineering. Morgan Kaufmann, Amsterdam
Molléri JS, Petersen K, Mendes E (2019) Cerse-catalog for empirical research in software
engineering: a systematic mapping study. Inf Softw Technol 105:117–149
Münch J, Schmid K (2013) Perspectives on the future of software engineering: essays in honor of
Dieter Rombach. Springer, Berlin
Myers GJ (1978) A controlled experiment in program testing and code walkthroughs/inspections.
Commun Assoc Comput Mach 21(9):760–768
Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density.
In: Proceedings of the 27th international conference on software engineering. ACM, New York,
pp 284–292
Ogborn J, Miller R (1994) Computational issues in modelling. The Falmer Press, Basingstoke
Ostrand TJ, Weyuker EJ, Bell RM (2004) Where the bugs are. In: ACM SIGSOFT software
engineering notes, vol 29. ACM, New York, pp 86–96
The Evolution of Empirical Methods in Software Engineering 23

Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software

engineering. In: Ease, vol 8, pp 68–77
Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping
studies in software engineering: an update. Inf Softw Technol 64:1–18
Pfleeger SL (1995) Experimental design and analysis in software engineering. Ann Softw Eng
1(1):219–253
Pfleeger SL, Kitchenham BA (2001) Principles of survey research: part 1: turning lemons into
lemonade. ACM SIGSOFT Softw Eng Notes 26(6):16–18
Porter AA, Selby RW (1990) Empirically guided software development using metric-based
classification trees. IEEE Softw 7(2):46–54
Rombach HD, Basili VR, Selby RW (1993) Experimental software engineering issues: critical
assessment and future directions. In: Proceedings of international workshop, Dagstuhl Castle,
September 14–18, 1992, vol 706. Springer, Berlin
Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software
engineering. Empir Softw Eng 14(2):131
Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering. In:
Guidelines and examples. Wiley, London
Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans
Softw Eng 25(4):557–572
Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental designs for
generalized causal inference. Mifflin and Company, Boston, MA
Sharp H, Dittrich Y, De Souza CR (2016) The role of ethnographic studies in empirical software
engineering. IEEE Trans Softw Eng 42(8):786–804
Shneiderman B, Mayer R, McKay D, Heller P (1977) Experimental investigations of the utility of
detailed flowcharts in programming. Commun Assoc Comput Mach 20(6):373–381
Shull F, Carver J, Travassos GH (2001) An empirical methodology for introducing software
processes. In: ACM SIGSOFT software engineering notes, vol 26. ACM, New York, pp 288–
296
Shull F, Mendoncça MG, Basili V, Carver J, Maldonado JC, Fabbri S, Travassos GH, Ferreira
MC (2004) Knowledge-sharing issues in experimental software engineering. Empir Softw Eng
9(1–2):111–137
Shull F, Singer J, Sjøberg DI (2007) Guide to advanced empirical software engineering. Springer,
Berlin
Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software
engineering. Empir Softw Eng 13(2):211–218
Sjøberg DI, Dybå T, Jorgensen M (2007) The future of empirical methods in software engineering
research. In: 2007 Future of software engineering. IEEE Computer Society, Silver Spring, pp
358–378
Srinivasan K, Fisher D (1995) Machine learning approaches to estimating software development
effort. IEEE Trans Softw Eng 21(2):126–137
Staron M (2019) Action research in software engineering: theory and applications. Springer, Berlin
Stol KJ, Fitzgerald B (2015) Theory-oriented software engineering. Sci Comput Program 101:79–
98
Stol KJ, Fitzgerald B (2018) The ABC of software engineering research. ACM Trans Softw Eng
Methodol 27(3):11
Stol KJ, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: a
critical review and guidelines. In: 2016 IEEE/ACM 38th international conference on software
engineering (ICSE). IEEE, Piscataway, pp 120–131
Swanson EB, Beath CM (1988) The use of case study data in software management research. J
Syst Softw 8(1):63–71
Theisen C, Dunaiski M, Williams L, Visser W (2017) Writing good software engineering research
papers: revisited. In: Proceedings of the 39th international conference on software engineering
companion. IEEE, Piscataway, pp 402–402
24 M. Felderer and G. H. Travassos

Thomke SH (2003) Experimentation matters: unlocking the potential of new technologies for
innovation. Harvard Business Press, Boston
Tian J (1995) Integrating time domain and input domain analyses of software reliability using
tree-based models. IEEE Trans Softw Eng 21(12):945–958
Travassos GH, Barros MO (2003) Contributions of in virtuo and in silico experiments for the
future of empirical studies in software engineering. In: 2nd workshop on empirical software
engineering the future of empirical studies in software engineering, pp 117–130
Travassos GH, dos Santos PSM, Mian PG, Neto ACD, Biolchini J (2008) An environment
to support large scale experimentation in software engineering. In: 13th IEEE international
conference on engineering of complex computer systems (ICECCS 2008). IEEE, Piscataway,
pp 193–202
Wagner S, Fernández DM, Felderer M, Vetrò A, Kalinowski M, Wieringa R, Pfahl D, Conte T,
Christiansson MT, Greer D, et al (2019) Status quo in requirements engineering: a theory and
a global family of surveys. ACM Trans Softw Eng Methodol 28(2):9
Widman L (1989) Expert system reasoning about dynamic systems by semi-quantitative simula-
tion. Comput Methods Prog Biomed Artif Intell Med 6(3):229–247
Wieringa R (2014a) Empirical research methods for technology validation: scaling up to practice.
J Syst Softw 95:19–31
Wieringa RJ (2014b) Design science methodology for information systems and software engineer-
ing. Springer, Berlin
Wohlin C, Aurum A (2015) Towards a decision-making structure for selecting a research design in
empirical software engineering. Empir Softw Eng 20(6):1427–1455
Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2000) Experimentation in
software engineering: an introduction. Kluwer Academic Publishers, Norwell, MA
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in
software engineering. Springer, Berlin
Zelkowitz MV, Wallace DR (1998) Experimental models for validating technology. Computer
31(5):23–31
Zendler A (2001) A preliminary software engineering theory as investigated by published
experiments. Empir Softw Eng 6(2):161–180
Zimmermann T, Zeller A, Weissgerber P, Diehl S (2005) Mining version histories to guide software
changes. IEEE Trans Softw Eng 31(6):429–445
 Part I
Study Strategies
Guidelines for Conducting Software
Engineering Research

Klaas-Jan Stol and Brian Fitzgerald

Abstract This chapter presents a holistic overview of software engineering

research strategies. It identifies the two main modes of research within the
software engineering research field, namely knowledge-seeking and solution-
seeking research—the Design Science model corresponding well with the latter.
We present the ABC framework for research strategies as a model to structure
knowledge-seeking research. The ABC represents three desirable aspects of
research—generalizability over actors (A), precise control of behavior (B), and
realism of context (C). Unfortunately, as our framework illustrates, these three
aspects cannot be simultaneously maximized. We describe the two dimensions
that provide the foundation of the ABC framework—generalizability and control,
explain the four different types of settings in which software engineering research
is conducted, and position eight archetypal research strategies within the ABC
framework. We illustrate each strategy with examples, identify appropriate
metaphors, and present an example of how the ABC framework can be used to
design a research program.

1 Introduction

Research methodology—the study of research methods—is receiving increasing

attention from software engineering (SE) researchers. Numerous books and papers
have been written on the topic (Easterbrook et al. 2008; Glass et al. 2002; Seaman
1999; Singer et al. 2000; Stol et al. 2016b; Wohlin et al. 2012). While these are very

K.-J. Stol ()

Lero—The Irish Software Research Centre and School of Computer Science and Information
Technology, University College Cork, Cork, Ireland
e-mail: [email protected]
B. Fitzgerald
Lero—The Irish Software Research Centre and Department of Computer Science and
Information Systems, University of Limerick, Limerick, Ireland
e-mail: [email protected]

© Springer Nature Switzerland AG 2020 27

M. Felderer, G. H. Travassos (eds.), Contemporary Empirical Methods in Software
Engineering, https://doi.org/10.1007/978-3-030-32489-6_2
28 K.-J. Stol and B. Fitzgerald

useful reference works, there are several issues with the current state of literature on
methodology. First, there is a strong emphasis on a limited set of specific methods,
in particular experimentation, case studies, and survey studies. Although these are
the three most used empirical methods (Stol and Fitzgerald 2018), many other
methods exist that have not received the same level of attention. A second issue
is that the field has no agreement on an overall taxonomy of methods, which is
somewhat problematic as methods vary in terms of granularity and scope. This
makes a systematic comparison of methods very challenging. Furthermore, new
methods are being adopted from other fields. Grounded theory, for example, has
gained widespread adoption within the SE literature in the last 15 years or so
(Stol et al. 2016b). (We note that, like many other methods used in SE, grounded
theory is not a “new” method, as it was developed in the 1960s by social scientists
Glaser and Strauss—however, its application is relatively new to the SE domain.)
Other techniques and methods that are relatively new to the software engineering
field include the repertory grid technique (Edwards et al. 2009) and ethnography
(Sharp et al. 2016). With new methods and techniques being adopted regularly, it
becomes challenging to understand how these new methods compare to established
approaches. Further, numerous sources present a range of research methods, but
these presentations are limited to “shopping lists” of methods: definitions without
a systematic comparison. Rather than maintaining a list of definitions of research
methods, a more systematic approach is needed that allows us to reason and position
existing methods, and new methods as they emerge. Hence, in this chapter we
present a taxonomy of research strategies.
There is an additional challenge within the software engineering research
community. Different methods have varying strengths and drawbacks, but it is
quite common to see unreasonable critiques of studies due to the research methods
employed. For example, a common complaint in reviews of case studies is that they
do not allow statistical generalizability. Similarly, experiments are often critiqued
on the basis that they involved computer science students solving “toy” problems,
thus rendering them unrealistic, and therefore not worthy of publication. Not
unreasonably, researchers may wonder which method, then, is the silver bullet that
can address all of these limitations?
The answer is none.
Instead of discussing research methods, we raise the level of abstraction and
have adopted the term research strategy. A research strategy can be considered a
category of research methods that have similar trade-offs in terms of generalizability
and the level of obtrusiveness or control of the research context—we return to these
two dimensions in a later section in this chapter. Previously, we outlined what we
have termed the ABC framework of research strategies and demonstrated how this
taxonomy is suitable for software engineering research (Stol and Fitzgerald 2018).
In this chapter we draw on this earlier work, elaborate on how the ABC framework
is related to Design Science, and provide general guidance for researchers to select
appropriate research strategies.
The remainder of this chapter is organized as follows. Section 2 starts with a
discussion of research goals, dimensions, and settings. This section presents the
Guidelines for Conducting Software Engineering Research 29

two modes of research, namely knowledge-seeking and solution-seeking research.

It outlines the ABC framework and positions it in relation to Design Science.
Section 3 discusses the eight archetypal research strategies that are represented
within the ABC framework. For each strategy, we discuss the essence of the strategy,
identify a metaphor for the strategy, and provide high-level guidelines. Because
research studies are never conducted in isolation, we discuss in Sect. 4 how the
ABC framework can be used to design research programs. Section 5 offers a list
of recommended readings. Finally, Sect. 6 concludes the chapter.

2 Foundations

This section introduces a number of concepts that together form the foundation
for the ABC framework. We first introduce the two modes of research in software
engineering: knowledge-seeking and solution-seeking research. These are two dis-
tinct modes representing different types of activities. This chapter focuses primarily
on one mode, namely knowledge-seeking research, but contrasts it with solution-
seeking research. In so doing, we draw a link to Design Science. Much has been
written on Design Science, which is why we do not discuss it in this chapter. Instead,
we refer interested readers to chapter “The Design Science Paradigm as a Frame for
Empirical Software Engineering” of this book.
We then return our attention to knowledge-seeking research and introduce
two key dimensions that are present in all knowledge-seeking studies: the level
of obtrusiveness and generalizability. Each research strategy represents a unique
combination along these two dimensions. This section ends with a discussion
of research settings, which refers to the environment in which the research is
conducted.

2.1 Knowledge-Seeking vs. Solution-Seeking Research

There are two modes of software engineering research: knowledge-seeking and

solution-seeking research. These two modes address different types of questions;
Wieringa (2009) has referred to these as knowledge questions and practical prob-
lems, respectively. Figure 1 presents how these two modes of research are positioned
within the wider context of SE research, the real world, and the SE knowledge base.
Knowledge-seeking studies aim to learn something about the world around us by
making observations in some type of environment—this includes the technologies,
organizations, and people in natural, contrived, or simulated (virtual) settings.
Knowledge-seeking research studies lead to new knowledge, which is typically
reported in research papers and books, thereby contributing to the software engi-
neering knowledge base, from which researchers may draw when designing new
studies.
30 K.-J. Stol and B. Fitzgerald

In solution-seeking studies, researchers design, create, or develop solutions for

a given software engineering challenge. The outcome of these studies includes
algorithms, models, and tools. Such solution-seeking studies may draw applicable
knowledge from the SE knowledge base, which might have originated in either
knowledge-seeking or solution-seeking research. Much research within the SE
domain is solution-seeking with resulting design artifacts. These artifacts represent
“design knowledge,” in that they embody knowledge on how a particular engineer-
ing problem can be solved—and this knowledge is added to the SE knowledge base
as well. Solution-seeking studies fit very well within a Design Science framework
(March and Smith 1995; Simon 1996), as discussed in more detail in chapter “The
Design Science Paradigm as a Frame for Empirical Software Engineering” of
this book. Implemented solutions can be deployed into the real world and their
effectiveness or utility can be studied using knowledge-seeking research. We note
that the research process for Design Science as proposed by Hevner et al. (2004)
does not align perfectly with solution-seeking research but claims a wider scope that
includes evaluation studies—we categorize the latter firmly as knowledge-seeking
studies.
As Wieringa (2009) has pointed out, knowledge-seeking and solution-seeking
research can be interlinked—nested, even—because knowledge is needed to design
solutions, and once designed, a researcher is interested in learning whether the solu-
tion works or how well it compares to other solutions. This linkage is represented
by the two white arrows in Fig. 1. In this chapter we are primarily concerned with
strategies to conduct knowledge-seeking research and refer readers interested in

Environment
Technology, Organizations, People

Observations Deployment of
Solutions

Software Knowledge- Solution-seeking

Engineering seeking research research
Research
ABC Framework Design Science

New Theoretical Applicable Design

knowledge knowledge knowledge knowledge

Software Engineering Knowledge base

Fig. 1 Knowledge-seeking and solution-seeking research: positioning the ABC framework and
Design Science
Guidelines for Conducting Software Engineering Research 31

Design Science to chapter “The Design Science Paradigm as a Frame for Empirical
Software Engineering”.

2.2 Two Dimensions of Research: Obtrusiveness

and Generalizability

In the remainder of this chapter we focus primarily on knowledge-seeking research.

Numerous methods can be used to “seek knowledge,” and as mentioned above, there
are numerous sources in the software engineering literature that provide lists of
methods. However, a systematic framework to position these methods in relation
to one another has been lacking. To address this, we draw on McGrath (1981,
1984, 1995), who organized the most common methods in the social sciences
into a methodological “circumplex” that positions eight research strategies. We
operationalized the circumplex for a software engineering context and have labeled
the result the “ABC framework” for reasons that will become clear. Below we
explain the key concepts of this framework.
The framework is organized along two dimensions: obtrusiveness and generaliz-
ability (see Fig. 2). The first dimension is concerned with how obtrusive the research

More obtrusive research

Increasingly Increasingly
more more specific
universal contexts and
contexts and systems
systems

Less obtrusive research

Fig. 2 Two dimensions in knowledge-seeking research

32 K.-J. Stol and B. Fitzgerald

is: to what extent does a researcher “intrude” on the research setting, or simply make
observations in an unobtrusive way. Research methods can vary considerably in the
level of intrusion and resulting level of “control” over the research setting. Clearly, a
study that seeks to evaluate the efficiency or performance of a tool requires a careful
study set-up, whereas a case study that seeks to describe how agile methods are
tailored at one specific company does not (Fitzgerald et al. 2013).
The second dimension is the level of generalizability of research findings. This
is a recurring concern in software engineering research, in particular in the context
of case studies. Indeed, exploratory case studies, and other types of field studies, are
limited in that the researcher cannot draw any statistically generalizable conclusions
from such studies. However, generalization of findings is not the goal of such
studies—instead, exploratory case studies and other types of field studies aim to
develop an understanding rather than generalization of findings across different
settings. Exploratory case studies can be used to theorize and propose hypotheses
about other similar contexts.
It is worth noting that a broader view of generalizability beyond that of the
statistical sample-based one has also been identified (e.g., Yin 2014; Lee and
Baskerville 2003). Yin identifies Level 1 inference generalizability which has two
forms. The first is the widely known statistical generalizability from a representative
sample to a population. He also identifies another Level 1 inference, namely from
experimental subjects to experimental findings, which is also quite relevant to
our research strategies. However, Yin also suggests a further Level 2 inference
category of analytic generalizability which involves generalizing to theory. This
could involve generalizing from a sample to a population, or, indeed, generalizing
from field study findings or experimental findings.

2.3 Research Settings

Research takes place in different settings, that is, the environment or context within
a researcher conducts research. McGrath (1984) identified four different types of
settings to conduct research. Building on the two dimensions described above, these
settings are positioned as four quadrants at a 45◦ angle with the main axes that
represent the two dimensions described above (see Fig. 3).
The first type of settings is natural settings, represented as Quadrant I. Natural
settings are those that naturally occur in the “field” and that exist independently from
the researcher conducting research; that is, settings that are host to the phenomenon
that a researcher wishes to study. For example, the “field” for a study on software
process improvement is likely to be a software development organization, whereas
the “field” can also be the online communication channels when the topic of study is
a particular open source software development project (Mockus et al. 2000)—after
all, for open source developers, these online channels are the (virtual) place where
they communicate and do work. Natural settings are always specific and concrete,
rather than abstract and general; hence, the quadrant representing natural settings is
Guidelines for Conducting Software Engineering Research 33

More obtrusive research

Quadrant II
Contrived Settings

Natural Settings
Neutral Settings

Quadrant I
Quadrant IV

Increasingly Increasingly
more more specific
universal contexts and
contexts and systems
systems

Quadrant IV
Non-empirical Settings

Less obtrusive research

Fig. 3 Research settings in knowledge-seeking research

positioned on the right-hand side of Fig. 3. Researchers may still exert some level
of control over a natural setting (Quadrant I, above the x axis) or may simply make
empirical observations without manipulating the research setting (below the x axis).
In contrast to natural settings, contrived settings (represented as Quadrant II) are
created by a researcher for the study. In a software engineering research context,
contrived settings include laboratories with specific and dedicated equipment to
conduct an experiment on some algorithm or software tool. Contrived settings are
characterized by a significant degree of control by the researcher. This manifests as
the set-up of specialized equipment and measurement instruments that facilitate the
execution of a study. Many experimental studies within software engineering are
conducted in such contrived settings, whereby algorithms and tools are evaluated
for performance and precision. Contrived settings are created by a researcher to
either mimic some specific or concrete class of systems (the right-hand side of
Quadrant II), or a more abstract and generic class of systems (the left-hand side
of Quadrant II). Either way, a contrived setting is always specifically set up by a
researcher, implying the researcher has a high degree of control over the research—
hence, Quadrant II is positioned at the upper half of the x axis. A contrived setting is
essential to conduct the research—without measurement instruments and other tools
34 K.-J. Stol and B. Fitzgerald

such as the design of scenarios or tasks for human participants, the study could not
be performed.
There are, however, also studies that do not rely on a specific setting. Some
types of studies can take place in any setting, and so the setting is neutral—this
is represented in Fig. 3 as Quadrant III. Researchers may or may not manipulate the
research setting; in any case, because the research setting is neutral and not specific
to any concrete or specific instance, Quadrant III is positioned at the left-hand side
of the x axis.
Finally, the fourth type of setting is non-empirical, represented by Quadrant IV
in Fig. 3. That is, this type of research does not lead to any empirical observations.
Within software engineering, non-empirical research includes the development of
conceptualizations or theoretical frameworks, and computer simulations. While
software engineering as a field of study has not traditionally been strongly focused
on the development of theory, several initiatives have emerged in recent years to
address this (Stol and Fitzgerald 2015; Ralph 2015; Wohlin et al. 2015). Quadrant IV
is positioned at the bottom of Fig. 3 because the researcher does not “intrude” on any
empirical setting. Non-empirical research is typically conducted at the researcher’s
desk or in his or her computer, through the development of symbolic models and
computer programs that mimic real settings.

2.4 The ABC of Software Engineering Research

Having laid the foundations for the ABC framework, we now populate the grid
in Fig. 3 with eight archetypal research strategies. The result is what we have
termed the ABC framework (see Fig. 4). Several of the research strategies will sound
familiar; for example, field study, laboratory experiment, and sample study, which
includes survey studies. Other terms such as experimental simulation may be less
known within the SE field. Section 3 presents each of these eight strategies in detail.
We now turn our attention to the last aspect of the framework, which are the markers
A, B, and C.
The term “ABC” seeks to convey the fact that knowledge-seeking research
generally involves actors (A) engaging in behavior (B) in a particular context (C).
Within software engineering, actors include software developers, users, managers,
and when seeking to generalize over a “population,” can also include non-human
artifacts such as software systems, tools, and prototypes. Behavior can relate to
that of software engineers, such as coordination, productivity, motivation, and
also system behavior (typically involving quality attributes such as reliability and
performance). Context can involve industrial settings within organizations, open
source communities, or even classroom or laboratory settings.
In the context of our discussion on obtrusiveness and generalizability above,
researchers will want to maximize the generalizability of the evidence across
actor populations (A), while also exercising precise measurement and control over
behavior (B) being studied, in as realistic a context (C) as possible. However,
Guidelines for Conducting Software Engineering Research 35

Maximum
More
potential for Quadrant II
obtrusive
precision of Contrived research settings
research
measurement of
B
Laboratory Experimental
Neutral research settings Experiments Simulations

Natural research settings

Judgment Field
Quadrant III

Studies Experiments

Quadrant I
Sample Field
Studies Studies

C
Formal Computer Maximum potential for
Theory Simulations realism of Context
Maximum potential for
generalizability over Actors
A

Less Quadrant IV
obtrusive
Non-empirical research settings
research

Increasingly more universal Increasingly more specific

contexts and systems contexts and systems

Fig. 4 The ABC framework positions eight archetypal research strategies along two dimensions:
generalizability of findings and obtrusiveness of the research (adapted from McGrath 1984)

as McGrath (1981) pointed out, it is impossible to maximize all three goals

simultaneously. Increasing precision of measurement and control of behavior (B),
for example, inevitably intrudes on and reduces the naturalness and thus the realism
of the context (C). Conversely, if one seeks to preserve the realism of context
(C), this will reduce both the precision of measurement of behavior (B) and also
the degree of generalizability over actors (A). This is reflected in Fig. 4 which
identifies the research strategies that are best positioned to deliver for each of the
A, B, and C. Sample studies can achieve high generalizability (A) but they sacrifice
realism of context (C) and precision of behavior (B). Laboratory experiments allow
precise measurement and control of behavior (B) but this comes at the expense
of the realism of context (C) and generalizability (A). Field studies maximize the
realism of context (C) but this is at the expense of control of behavior (B) and
generalizability (A). Clearly, the full range of research strategies is required to
deliver across all three research goals and these need to be planned and managed.
The above also highlights the fact that certain strategies have inherent and intrinsic
weaknesses which cannot be overcome—thus field studies can never provide
generalizability, but that is neither their purpose nor strength, and this is not a
36 K.-J. Stol and B. Fitzgerald

limitation which can be overcome when using this research strategy. Therefore,
research studies adopting that strategy should not be criticized on that basis.

3 Strategies for Software Engineering Research

In this section we outline the eight archetypal research strategies that are positioned
in the ABC framework in Fig. 4. We discuss the research strategies as organized by
the quadrants discussed in Sect. 2.3, starting in Quadrant I. Table 1 summarizes
the discussion for each strategy, including a metaphor that might help in better
understanding the nature and essence of the research strategy, how that setting
manifests in software engineering research, and general suggestions as to when to
use that strategy.

3.1 Field Studies

Field studies are conducted in natural settings; that is, settings that pre-exist the
design of the research study. Field studies are best suited for studying specific
instances of phenomena, events, people or teams, and systems. This type of
research helps researchers to understand “what is going on,” “how things work,”
and tends to lead to descriptive and exploratory insights. Such descriptions are
useful because they provide empirical evidence of phenomena that are relevant
to software engineering practitioners, students, and researchers. The findings may
provide the basis for hypotheses, which can then be further studied using other
strategies. Typical examples in software engineering research are the case studies of
the Apache web-server (Mockus et al. 2000) and agile method tailoring (Fitzgerald
et al. 2013).
Field studies are relatively unobtrusive with respect to the research setting. The
setting for field studies is akin to a jungle, a natural setting that contains unexplained
phenomena, unknown tribes, and secrets that the researcher seeks to discover and
understand (see Fig. 5). Within a software engineering context, the researcher does
not manipulate the research setting, but merely collects data to describe and develop
an understanding of a phenomenon, a specific system, or a specific development
team. This is why field studies are best suited to offer a high degree of realism of
context as the researcher studies a phenomenon within its natural setting, and not
one that the researcher manipulated.
Typical research methods include the descriptive or exploratory case study, and
ethnography (Sharp et al. 2016), but archival studies of legacy systems also fall
within the category of field studies, for example, Spinellis and Avgeriou’s study
of the evolution of Unix’s system architecture (Spinellis and Avgeriou 2019). An
alternative metaphor for such archival studies is an archaeological site rather than
a jungle. Data collection methods for field studies include (but are not limited to)
Table 1 Research strategies in software engineering
Strategy Metaphor Setting in SE When to use
Field study Jungle: a natural setting that is ideally left Software engineering phenomena in a natural To understand phenomena: How does it work?
untouched, where creatures and plants can be context, such as pair programming in industry, open how and why do project teams do what they do?
observed in the wild with a great level of detail source software projects, etc. what are characteristics of a phenomenon?
Maximum potential to capture a realistic context
Field Nature reserve: a natural setting that has some level Industry or open source software projects or teams To measure “effects” of some intervention in a
experiment of manipulation, e.g., fences, barriers, closed-off with some level of researcher intervention; natural setting, acknowledging lack of precision
sections, sections treated with some intervention interventions could include different workflows, due to confounding factors that cannot be
tools controlled for
Experimental Flight simulator: a contrived environment to let Realism varies from classroom to industry settings To evaluate/measure behavior of participants on a
simulation pilots train specifically programmed scenarios to designed by researchers with a specific set of tasks set of tasks in a setting that seeks to resemble a
evaluate their behavior and decisions. Realism or scenarios which recruited participants are asked real-world setting
varies depending on resources to process
Laboratory Cleanroom/test tube: highly controlled setting Classroom or research laboratory settings with a To make high-precision measurements, e.g., for
experiment allowing a researcher to make measurements with specific set-up and instrumentation to measure, e.g., comparing different algorithms and tools.
high degree of precision performance of algorithms or tools Maximum potential for precision of measurement
Judgment Courtroom: neutral setting to present Online or offline setting to solicit input from To get input (“judgment”) from experts on a
study evidence/exhibits to a carefully selected panel, carefully selected experts after presenting them withgiven topic, which requires intense
asking them for a response (e.g., guilty) a question on a topic or an exhibit (e.g., a new tool)
stimulus/response communication
Guidelines for Conducting Software Engineering Research

Sample study Referendum: a process to collect a sample of data to Online surveys conducted among a population of To answer generalizability questions, incl.
seek generalizability over the population. Unusable (typically) developers, or data collected from a characterization of a dataset, correlation studies.
(invalid) data must be filtered out before analysis software repository. Data must be checked before Maximum potential for generalizability over
analysis findings
Formal Jigsaw puzzle: attempt to make sense of, integrate, Given a set of related observations and evidence To provide a framework that can describe,
theory or fit in different pieces into a coherent “picture” regarding a topic of interest, aim to find common explain, or predict phenomena or events of
patterns and codify these as a theory interest, while remaining consistent
(generalizable) across different events within
some boundary
Computer Weather forecasting system: model of the real A computer program that simulates a real-world To develop an understanding of phenomena and
simulation world is programmed, capturing as many phenomenon, capturing as many important settings that are too complex or expensive to
parameters as possible. Scenarios are run to make parameters as possible. Program different scenarios create in the real world
informed predictions, but cannot anticipate events to “run” to explore ranges of, and interactions
not programmed in the simulation between, parameters of interest
37
38 K.-J. Stol and B. Fitzgerald

Fig. 5 Field studies are conducted in pre-existing settings that are not manipulated by a researcher,
to study and observe natural phenomena and actors. Image credits: Public domain. Source:
maxpixel.net (no date)

interviews, document and archival study, or mining repositories—Lethbridge et al.

(2005) have discussed a variety of data collection methods and their trade-offs for
field studies.

Guidelines for Field Studies

• Use the field study strategy to study phenomena in their natural setting, to
understand “what is going on,” or “how things work.” They provide good
opportunities to develop substantive theory. Typical methods include the
exploratory case study, ethnography, and archival study.
• Field studies require a high level of attention and engagement with the
subject and setting. Audio and video recording may help to capture details
for later analysis, but these media may affect the behavior of human actors.
• Success of field studies depends on good access to the relevant people and
artifacts, which can be particularly challenging within corporate settings.
An internal “champion” or maintaining good relationships is key.
• To generalize findings from field studies, use complementary strategies
such as formal theory and sample studies.
Guidelines for Conducting Software Engineering Research 39

3.2 Field Experiments

Field experiments are also conducted in natural settings, but unlike field studies,
this type of study involves some type of manipulation, thus imposing a greater
degree of control. That is, the researcher introduces some form of experimental
set-up by making changes to some variables of interest. After making such changes,
the researcher may observe some effect. If field studies are conducted in a “jungle,”
then the setting for a field experiment is more like a nature reserve: a dedicated
area that may be very similar to a jungle, but the researcher can introduce specific
changes to study different aspects within the reserve. Figure 6 shows a jungle with
specific patches of trees cut down; the purpose of this study was to study the effects
of different types of forest fragmentation on wind dynamics and seed dispersal
(Damschen et al. 2014).
It is important to remember that field experiments take place in natural settings,
which should be clearly distinguished from contrived settings that provide the
setting for experimental simulations and laboratory experiments, discussed later.
A range of methods are available to conduct field experiments. These are not
limited to the traditional controlled experiment that separates a population of actors
into two or more different groups so as to make comparisons. Experimentation also
occurs when a researcher adopts the action research method; with action research, a
researcher follows a recurring cycle of making changes and evaluating the results of
those changes. While not a randomized controlled experiment, action research can
still be considered a form of experimentation.
Ebert et al. (2001) conducted a field experiment to investigate three factors that
might impact the cost of rework in distributed software development: (1) the effect

Fig. 6 Field experiments

involve the manipulation of
an otherwise pre-existing Wing
natural setting to facilitate
observation and measurement Unconnected
in order to collect data. Image winged patch
source: Damschen et al.
Connected
(2014), used with permission
patch

270° 90°

Corridor
Unconnected
rectangle patch
Unconnected
winged patch
40 K.-J. Stol and B. Fitzgerald

of co-location on the efficiency and effectiveness of defect detection; (2) the effect
of coaching on software quality, and (3) the effect of changes to the development
process on teamwork, and continuous build on management of distributed project.
To evaluate these effects, Ebert et al. used project data that the company had
gathered for several years.
Despite careful measurement of a range of parameters, certain factors are hard
to measure, such as “culture.” Ebert et al. divided the projects into different sets,
e.g., “within one culture (i.e., Europe).” While we can certainly generalize that
there are common attributes across different European cultures, there is no single
Europe culture—each European culture is quite distinct, with significant changes
even between neighboring countries such as Ireland and the UK, or the Netherlands
and Germany. Furthermore, each of the projects in the data set used by Ebert et
al. will undoubtedly have had specific obstacles, such as particularly challenging
technology or changing requirements, and strengths such as particularly talented
staff. These factors are very hard, if not impossible to capture, reducing the precision
of measurement.

Guidelines for Field Experiments

• Use the field experiment strategy to evaluate the effects of manipulations
within realistic settings.
• Field experiments require a high degree of prior investment in design and
execution of data collection, typically in corporate settings.
• Contextual factors must be recorded carefully, so that they can be consid-
ered in the analysis.
• Use formal theory or sample studies to seek a higher degree of generaliz-
ability. Computer simulations can be used to model a complex real-world
system to explore key parameters and their interactions when a field
experiment would be too costly.

3.3 Experimental Simulations

Experimental simulations combine some elements of the field experiment strategy

and laboratory experiments (discussed below). A key difference with field experi-
ments is that experimental simulations take place in contrived settings. That is, the
research environment is artificial and is purposely designed to conduct the study—
before and beyond the study, it does not exist. While this makes the experimental
simulation less realistic, it also gives the researcher opportunities to make more
precise measurements and observations, because participants can be asked to do
specifically designed tasks that may not be part of their daily routine. This increases
the level of control even further with respect to field experiments. While we
compared the field experiment to a nature reserve, the experimental simulation is
Guidelines for Conducting Software Engineering Research 41

Fig. 7 Flight simulators are experimental simulations that facilitate training and study of the
behavior of pilots in pre-programmed scenarios. Image credits: SuperJet International, distributed
under CC BY-SA 2.0. Source: SuperJet International (2011)

more akin to a greenhouse, which mimics a warmer climate. The researcher is still
interested in natural processes (e.g., how do flora flourish), but the setting in which
that process is observed is artificially created for that purpose.
The greenhouse metaphor links well to the jungle and nature reserve metaphors,
but another useful metaphor for the experimental simulation is a flight simulator (see
Fig. 7). Within the flight simulator, specific events can be introduced, such as heavy
storms and rainfall. Pilots in training would be asked to perform as they would in a
real aircraft, but the research setting is considerably easier and cheaper to plan.
The level of realism that is achieved in experimental simulations can vary
considerably, just like flight simulators. The latter can vary from low-cost set-ups
consisting of a standard PC, a budget flight yoke, and rudder pedals to high-
end, full motion flight simulators used by professional pilots that might cost
millions of dollars. The tasks that participants are asked to perform in experimental
simulation may also vary in realism. While such tasks are part of normal daily life
of the participants in field experiments, in experimental simulations participants
are recruited and invited to perform certain tasks designed by the researcher. The
process that the researcher wishes to study is simulated, facilitating systematic
measurement and comparison. The level of realism of the task can be very high,
such as debugging a program within a professional setting (Jiang et al. 2017), or it
can be as contrived as producing and trading colored shapes, such as blue squares
and red circles (Bos et al. 2004).
42 K.-J. Stol and B. Fitzgerald

Jiang et al. (2017) conducted an experimental simulation to investigate whether

developers conduct impact analysis during debugging. Their contrived setting
consisted of a specifically set-up work station, equipped with the SimpleScreen-
Recorder recording software. Videos were captured of nine professional developers
who had been given two bug reports. The bug reports had been identified by the
researchers in two specific applications (PdfSam and Raptor) and were selected
because these bugs had already been fixed at the time of the study. While the task at
hand and the study setting were contrived (designed by the researchers), both were
quite realistic. However, the realism of the study was reduced as the researchers set
a time limit for some of the participants and offered a suggested fix to the defects.
In contrast, Bos et al. (2004) focused on collaboration between co-located and
distributed individuals, and the task at hand was simply a mechanism to engage
participants. The focus of the study was therefore on the emergent behavior of the
participants, rather than how a specific task was performed as was the case for
the Jiang et al. study described above. Therefore, to ensure that the participants
understood the task at hand, and to minimize any distraction that might ensue from
introducing complex tasks, the researchers chose to design trivial tasks.

Guidelines for Experimental Simulations

• Use experimental simulations to find an appropriate balance between
measuring emergent behavior and achieving realism.
• Choose an appropriate level of realism of the research setting, balancing
between cost and effort of setting up the research setting and the need for
realism.
• If the research focus is on participant behavior that is not dependent on a
specific type of task, then minimize the complexity of the task.
• If the research focus is on the behavior in relation to a specific type of task,
then the task should exhibit a high level of realism.

3.4 Laboratory Experiments

The laboratory experiment is a strategy that offers maximum opportunity to control

the research setting. Like experimental simulations, laboratory experiments take
place in contrived settings that are created by the researcher. Laboratory experiments
involve a careful manipulation or initialization of variables and settings that give
the researcher the maximum opportunity to take precise measurements of actors’
behavior, whether these are human participants or software systems.
If the experimental simulation is akin to a greenhouse, then we can compare the
laboratory experiment to a test tube or cleanroom (see Fig. 8). Both a greenhouse
and a test tube provide a contrived setting, but the difference lies in the compromise
Exploring the Variety of Random
Documents with Different Content
 “We’ll get out of these rocks some time this morning,” predicted
Mr. Lewis with a smile.
And he was right. Gradually the boulders they passed grew
smaller and the soil more loose. By the time they had stopped for
the noon meal they were again among sand dunes.
The heat was now terrific. If it had been warm before, it was
scorching now. Everywhere they went they were under the blaze of
the fierce sun. How the camels managed to keep from burning their
feet was a mystery to the youths.
Their throats were parched, their tongues numb. Water, water!
If they could only drink and drink and drink! But only small amounts
were allowed to be taken, for this region was many, many miles
across, and there was no well or oasis anywhere near their path of
traveling.
“If we have much more of this I’m afraid I’ll fall off my camel,”
said Bob with a grim smile.
“Not quite that bad off, are you?” laughed Dr. Kirshner. “Dying of
thirst is a rare occurrence in this part of the Sahara. But it does
happen sometimes, and it is a tragic death indeed.”
“Worst thing is,” explained Mr. Holton, “there is a time when the
victim of thirst would die should he touch water. In that case, water
is virtually a poison.”
The sand hills that they were passing over were much lower
than those in the country below Wargla. The desert stretched away
to the horizon in endless waves, which, as far as the travelers could
see, were unbroken.
Vegetation was scarce, only a scattering of yellow plants dotting
the dunes. This promised to be a disadvantage to the dromedaries,
for previously they had occasionally nibbled on the trees and shrubs
that were clustered about.
“Look at the sky,” said Joe, turning his gaze upward.
“Funny color, isn’t it?” Bob returned. Then, as he peered into the
distance, he uttered an exclamation of surprise and fear.
But the others had seen also and were equally as excited.
 Away to their right a heavy mist had risen and was rapidly
turning reddish.
“A sandstorm!” cried Fekmah in great anxiety. “A sandstorm is
coming!”
 CHAPTER XVII
Moments of Horror

T HE explorers, particularly Tishmak, knew the danger of a

sandstorm. It was not infrequent for large caravans to be completely
engulfed by the heavy veil of sand, leaving only the dead bodies of
the camels and their riders. The Americans remembered a tale that
Fekmah had told them about a trading caravan of five hundred
dromedaries coming to a tragic end in this region. Would their little
caravan also perish?
“Get your goggles,” commanded Fekmah, his tone indicating
that he was calm even in the face of danger.
“And be sure they fit tightly!” warned Dr. Kirshner. “Even then
we’ll get some of the sand.”
The atmosphere was rapidly becoming extremely dry and hot,
and at intervals a fierce wind brought minute particles of sand into
the explorers’ faces.
“Now,” began Fekmah, after conversing briefly with the guide,
“we must get dromedaries in group, so they not get fright and run
away. Then we crouch down behind them.”
The camels were drawn up together and fastened in a circle
with ropes.
“It might be wise to put up our tent, mightn’t it?” asked Mr.
Lewis, but Fekmah shook his head vigorously.
“No, no,” he said. “Then we get in trap and not get out. If sand
very heavy, we want to be in open.”
The dense mist was thickening and spreading, until it soon
covered the whole horizon. The sky in the distance was not visible
for the heavy cloud of fine particles.
 The explorers got out blankets and wrapped themselves tightly.
Even then, said Fekmah, the small bits of soil would get through to
their skins.
They had scarcely finished preparations when the first breaker
suddenly came with all force, striking the adventurers in the face
and penetrating the blanket.
It was blinding, smothering, but they managed to get air and
fought with a determination that was born of adventure. Crouching
behind the sturdy dromedaries, they held their heads low to avoid as
much of the fury of the storm as possible.
It was with great difficulty that the camels kept their positions
together, but they succeeded admirably.
“Doesn’t seem right for them to have to stop the sand for us,”
said Joe, shouting in order to make himself heard.
“It’s a shame,” Bob shouted back. “But they can probably stand
it better than we can.”
Slowly they found themselves enveloped in a heavy opaque
atmosphere, so dense as to seem almost as a wall. The thought of
being completely covered up was constantly in their minds, bringing
about almost a feeling of despair.
The burning wind was constantly lashing them in the face, until
it seemed that they could stand it no longer. Indeed, if their heavy
goggles had not been of unbreakable glass, the furious particles of
sand would have smashed them in the explorers’ eyes. Even as it
was, some of the sand found its way in.
“This is terrible!” moaned Joe. “Awful—simply——”
He stopped suddenly, as his mouth became filled with sand.
Another gust of wind had come, bringing with it an enormous
quantity of the burning sand.
The explorers’ eyes were smarting, their lips were cracked and
bleeding. They felt that they would smother. Nothing could have
been worse, it seemed.
They could hear the dromedaries snorting with fear and
irritation. What if the brutes could not stand?
 Conversation was now impossible, for they dared not open their
mouths for fear of swallowing some of the stinging sand. Even when
they breathed, the fine particles filtered through the net that hung
over their faces.
The sky above was of a bright red color, and a weird light
trickled through the fog of yellow. It was the most unusual
happening that the Americans had ever witnessed.
“If it just wasn’t for this terrible wind!” muttered Mr. Holton,
when there had come a slight lull.
“Yes,” agreed Fekmah. “Then it not be so hard to stand it.”
He had scarcely finished when another gust of hot sand struck
them cruelly, making their faces sting anew.
Suddenly Tishmak noticed that they were nearly engulfed in a
heavy pile of sand. With a quick motion he drew himself out and
drove the dromedaries to another spot.
For a brief moment the explorers were exposed to the full
violence of the storm. Then they again took places behind the newly
located camels.
“Not taking any chances on being covered up, are you?” said Dr.
Kirshner to Tishmak.
The latter did not understand the words, but he caught the
meaning and smiled.
How long the terrific onslaught of sand lasted, no one knew.
They had lost all sense of time, and the heavy atmosphere
completely hid the sun.
It was only gradually that the terrible storm subsided, and then
the air was exceedingly hot and dry, promising to remain that way
for some time. Slowly the cloud of sand about them grew thin, until
it finally cleared away completely. Now only an occasional hot wind
annoyed them, but it was scarcely anything compared to the
previous bombardment of sand.
“No more of anything like that for me!” muttered Bob, as he
worked his feet loose from the high pile that strove to bury him
alive.
 The dromedaries, too, had their legs embedded in the sand so
deeply that it required several minutes of constant digging to relieve
them.
“Suppose we rest awhile before going on,” suggested Bob. “It
has been a great strain for all of us, standing against that terrible
rush of sand.”
The others readily agreed, and all thoughts of continuing the
journey at once were dismissed from mind.
“At least,” Joe said, “we got out alive, and that’s more than you
can say of many caravans.”
“Yes,” returned his father. “Perhaps under this very spot are the
bones of men and camels that were not as lucky as we were.”
“That storm rather short lasting,” remarked Fekmah, glancing at
his watch. “Many times storm last several hours.”
Joe sighed.
“I’d hate to have had to stand much more of it,” he said.
It was nearly noon, and the tent was pitched for the midday
rest. All were very weary after the terrific strain.
“Let’s have our lunch,” suggested Mr. Lewis. “I’m very hungry,
and I’m sure everyone else is.”
The noon meal and rest followed, the explorers not continuing
until after three o’clock.
Late that afternoon they came to one of the largest uninhabited
oases that they had yet seen. It was situated snugly on a narrow
stretch between high dunes.
“It’s a wonder a small town hasn’t sprung up around here,”
remarked Bob, drinking greedily of the refreshing water that gushed
from the large spring.
Dr. Kirshner nodded.
“With all these palm trees and the abundance of water it is
surprising,” he said. “But I suppose there are so few people, even
among the natives, who would live here that it wouldn’t pay.”
The containers were hurriedly filled.
 “It might be well to stay here for the night,” said Fekmah. “It is
getting late, and we all need sleep very bad.”
He turned to Tishmak and put the question before him in the
native language.
The guide at once gave his approval, more than glad of the
chance to stop.
“He say he wanted to stay here for night, but thought we in big
hurry,” Fekmah told the Americans.
“We are,” returned Mr. Holton. “But here is a very good place to
camp, and I think we’d better take advantage of the opportunity.”
The tent and provisions were unpacked from the camels, which
seemed more than glad of the chance to relax.
“Funny,” remarked Mr. Lewis, “that camels don’t care to lie in
the shade when there is an opportunity. You would think the terrible
sun would be avoided as much as possible, but that is not the case.”
“Either they like the heat or they are too lazy to move,” said
Joe.
For some time the two youths sat with their elders. Then Bob
got up and stretched.
“Suppose you and I get on our dromedaries and ride over to
that distant hill,” he said to Joe, pointing away to the horizon. “I’d
like to see what’s beyond there. This seems to be very high ground,
and we might get a view of the distant mountains from the top of
that dune.”
“Be sure and take your rifles, boys,” warned Mr. Lewis. “And
don’t stay too long.”
The boys slung their guns over their shoulders and rode off,
waving to their friends.
The hill that Bob referred to was at least a mile away, and the
ground on the way was of loose sand. The boys urged their mounts
to trot faster, however, and they would probably cover the distance
in a very short time.
 “I wonder if we could get a glimpse of the Ahaggar Mountains?”
said Joe.
“Might. But you must remember that we are still a great
distance away.”
As the boys had expected, they came to the hill in but a few
minutes. It was very high and steep, but the soil was hard. The
dromedaries had no difficulty in climbing steadily up.
At last they came to the top and gazed out into the distance.
“Look!” cried Joe. “The mountains! We can see them!”
Sure enough, the Ahaggar range was visible, stretching miles
and miles to either side. A few sharp peaks protruded high above
the others, but for the most part the line of mountains was rather
regular.
“Suppose that high peak is Illiman?” asked Joe, pointing to a
high crag that towered above the other mountains.
“You mean the one Fekmah was talking about? It might be. He
would know if he saw it, I suppose. And of course Tishmak would.”
The youths spent nearly a half-hour peering out at the
mountains, greatly impressed by the wonderful view.
“How far away do you suppose they are?” questioned Joe.
“Fifty miles, at least; maybe more. It will probably take us
another half-day to get to them.”
Finally the youths turned and rode back down the hill to tell
their elders of the magnificent view. Fekmah particularly would be
pleased, Joe thought.
But the boys were not overly anxious to get back to the oasis at
once. There were many other high sand dunes that they would like
to ride over.
“We won’t stay much longer,” said Bob. “Just ride around a bit.”
To their right was another high hill that might afford a view in
another direction. The youths rode over to it and climbed the
gradual side.
 Then, when they came to the top, they cried out in surprise and
fear.
In the distance appeared to be a whole regiment of galloping
horsemen coming toward them!
 CHAPTER XVIII
Savage Tribesmen

F OR a moment the youths were taken completely aback in

surprise. That they would see anything like this away out on the
Sahara was not in the least expected. They stood for some time in
sheer amazement and not a little fear.
“An army coming at us!” muttered Bob, staring at the distant
spectacle.
“An army, yes. Must be five hundred cavalrymen.”
“But—but it can’t be! It’s impossible. What would soldiers be
doing away out here on the desert? Something’s seriously wrong
somewhere. If just one of us should see such a thing it might
indicate that the old brain wasn’t working just right, but for you and
I both——”
“Come on,” suggested Joe, giving his dromedary a slight kick.
“Let’s get out of here. I’m greatly worried.”
The youths turned their camels back to camp for a short
distance; then they urged them on to a fast trot.
They were not a little relieved when they finally reached the
oasis, where they found their friends awaiting them.
“Where have you been so long?” inquired Mr. Lewis, his face not
a little serious.
“We thought maybe something held you back,” added Dr.
Kirshner.
“It did,” replied Bob, trying to remain calm.
The men sat up quickly, sensing that some misfortune had come
upon the boys.
 “What was it?” demanded Mr. Holton tensely.
“An army,” Joe returned soberly.
For a second there was silence. Then the men broke out in
laughter. Evidently they thought the youths were joking. Even
Fekmah joined in, his dark features drawn together in mirth.
“Nothing to laugh at,” said Joe, vexed because the men thought
their experience funny. “It nearly scared Bob and me out of our
wits.”
Mr. Holton grew more serious.
“Come, now,” he said. “Tell us what you mean.”
Joe told of seeing the phenomenon from the top of the hill,
saying that there appeared to be at least five hundred horsemen
coming toward them.
When he had finished, the naturalists and Dr. Kirshner jumped
up in wonder and not a little fear, but Fekmah only laughed.
“W-what’s humorous!” demanded Mr. Lewis, greatly perplexed.
“Everything,” said Fekmah, laughing still harder. “What the
young men saw was only an illusion or mirage. There no army on
Sahara. Only look like army.”
“You mean it was a trick of nature, like the more common
mirages of lakes on the desert?” asked Dr. Kirshner with great
interest.
“Yes,” the Arab answered. “Caused by the bending of the rays of
light when they strike the hot sand.”
“Well, that’s a new one on me!” confessed Bob. “I was aware of
the fact that mirages of lakes are common, but that I should see an
army——”
It was now rapidly becoming dark. The explorers thought it best
to sleep all through the night and not wait for the moon, for they
greatly needed the rest.
“Tomorrow morning I’d like to see that mirage that you boys
thought was an army,” said Mr. Holton, when they prepared to retire.
 “And I, too,” put in Dr. Kirshner. “As it isn’t out of our way, we
can all ride over there.”
“It’ll be a good chance to take some motion pictures,” said Bob.
“A scene as unusual as that is sure to attract the curiosity of an
audience.”
Tishmak informed them that they would be out of this short
sand stretch early the next morning. Then they would come into the
Ahaggar Mountains, the real home of the mysterious Tuaregs.
“And I expect to begin my work in this region,” announced Dr.
Kirshner. “Perhaps if I put legend and history together, I can locate
something that will prove of great value to the world of archæology.
I have in mind at present the tomb of a great king who reigned in
those mountains many thousands of years ago. He is said to be an
ancestor of the Berbers, who are related to the Tuaregs. When we
come to the many Tuareg villages, I intend to make inquiries as to
their ancient legends.”
They were up early the next morning, anticipating the
exploration of the mountains that lay ahead of them.
But in order to get to the Ahaggars, it would be necessary to
continue for a short distance over the sand dunes.
After breakfast they rode over to the distant hill to get a view of
the mirage seen by the boys the day before. Sure enough, the army
of horsemen appeared to be riding toward them, and the details
were rather plain.
Mr. Lewis shook his head in bewilderment.
“Sure is strange,” he muttered. “Why should the horses and the
riders be so clearly defined? I can easily understand the mirage of a
lake, but this sure gets my goat.”
They stood for some time staring at the distant spectacle, Bob
and Joe taking motion pictures. Finally they rode on up the hill to
catch a glimpse of the Ahaggars.
“I rather think that peak not Illiman but Oudane,” said Fekmah
to the youths, in answer to their question asking the name of the
distant high mountain. “Mount Oudane very high, and much nearer
than Mount Illiman.”
More movies were taken by the youths. Then they rode down
the opposite side of the dune in the direction of the mountains.
“Ahaggars very strange,” said Fekmah to the Americans, as they
rode in a group at the back of their pack camels. “There are high
cliffs, tall needle-like peaks, deep caves. There are canyons, ravines,
underground passageways. We see much, and we too be in great
danger.”
“Danger?” Joe looked up in some surprise.
“Yes. Very great danger. Wild Tuaregs roam about, and when on
a raid, think only of robbing travelers. Then, too, we be in region
where the two thieves who stole my map are. They perhaps be
waiting for us and shoot us quick without giving warning. Many
other dangers we might see.”
Fekmah sobered the Americans a little. They had not anticipated
any great peril, although they knew the two thieves might, should
they have arrived at the hidden riches first, give them trouble.
“But we’ll come out all right,” predicted Bob, again becoming
cheerful. “We’ll show those fellows that we’re capable of attending
to any crisis.”
A little farther on they reached the wall of rock that had
previously shut out the view of the mountains. It stretched many
miles to their right and left, but there were numerous breaks that
afforded openings into the country beyond.
They had barely reached the other side of the wall-like
formation when Joe caught sight of a group of tents quite a distance
to the east. He motioned for his friends to look in that direction.
“Probably Arabs,” pronounced Fekmah, after Tishmak had
chattered rapidly for a moment. “They nomads, who wander about
the desert taking their flock of goats with them.”
“Suppose we go over and see them,” suggested Mr. Holton.
“Perhaps they can give us a description of the country ahead of us.
There may be many more wells than we think, and it will do us no
harm to know of them.”
The others were in favor of carrying out Mr. Holton’s move. But
Fekmah warned them to be on the lookout for treachery.
“They probably not do us harm, but can never tell,” he said, as
the dromedaries were turned in the direction of the tents.
They reached the encampment in a very short time and were
about to look up some of the Arabs when a savage growl made
them wheel around in surprise and fear.
“Look!” cried Joe, laying his hand on his rifle.
Two large, savage dogs were making toward them with all fury,
showing their terrible teeth in anger. The enraged creatures were
probably owned by the Arabs in the tents and were acting as guards
against all marauders.
The foremost dog was almost upon Mr. Lewis’s camel. In
another moment the beast would sink its teeth in the dromedary’s
throat.
Displaying the quickness of a cat, the naturalist unslung his rifle,
took hasty aim, and fired.
The report of the gun was followed by a longdrawn howl from
the huge dog.
“Quick!” cried Bob. “The other dog!”
The second beast was rushing forward angrily.
Mr. Lewis again took aim. The others, trusting in his
marksmanship, made no move to get their rifles.
Click! There was no report this time. His magazine was empty!
Mr. Holton tried vainly to get his rifle out in time. Something
must be done at once, for the savage dog would be at the camels in
but a moment.
Suddenly Joe leaped from his camel directly in the path of the
oncoming animal. The dog stopped for a second, then rushed at the
youth with terrible ferocity.
 “It’s now or never!” Joe thought and brought the butt of his rifle
down with all his strength on the dog’s head.
There was a cry of pain, and the next moment the beast rolled
over in a dazed condition. At last the terrible enemies had been
overcome.
“Great work, Joe!” praised Mr. Holton. “We weren’t expecting to
see you act so quickly.”
“I didn’t know whether I could hit him at the right time or not,”
the youth said, wiping the perspiration from his brow. “But I thought
I’d take a chance. It——”
He stopped fearfully as a rifle shot rang out. Another report
followed the first, and Tishmak fell from his dromedary.
“Back!” cried Dr. Kirshner. “It’s the Arabs shooting from the
tents. Hurry or we’ll all be hit!”
Tishmak was rapidly picked up and placed on his camel, and
then the explorers retreated behind a formation of rock near the
high wall of stone that was to their right.
“You look after Tishmak,” said Mr. Holton to Dr. Kirshner.
“Meanwhile we’ll keep these Arabs away. We certainly aroused their
tempers when we put those dogs out.”
A volley of shots came from the Arabs’ tents, and the Americans
at once answered with their own rifles. Wherever a shot was heard,
Mr. Holton directed his friends to fire at the spot.
Suddenly Mr. Lewis caught a glimpse of a large one-armed Arab
who emerged into full view to send a bullet at his white enemies.
Without hesitation the naturalist fired, bringing the man down with a
thud.
“Look!” cried Bob. “They’re backing up. That fellow you shot
must have been the leader.”
“Does seem that way,” agreed Mr. Lewis. “But we must remain
on guard. These are treacherous characters.”
Only an occasional shot rang out. Then finally there was silence.
 “Now we’ll see how Tishmak is,” said Mr. Holton, leaving his
position at the end of the rocky crag.
They found that Dr. Kirshner had bound and treated the wound,
which was in the left arm. The Arab seemed in high spirits, despite
the fact that he was evidently in pain.
“It doesn’t appear serious,” said the archæologist. “With the
right kind of attention it will probably be all right in a few days.”
“Lucky that he wasn’t killed, or that more of us weren’t hit,”
remarked Mr. Lewis gravely. “The Sahara is a dangerous place for
explorers.”
They waited several minutes for any more rifle shots from the
Arabs, but none came. Finally Mr. Holton mounted his dromedary.
“Let’s get on our way,” he suggested. “I don’t think there’s any
danger now. The Arabs have retreated to a distance beyond their
tents, and I believe they’ll stay there awhile.”
Tishmak was helped on his camel. Then, when the others had
also mounted, they rode off.
They were now rapidly leaving the region of low sand dunes
behind. Rocks of all sizes and shapes became more numerous, and
vegetation was more abundant. There were, however, stretches of
coarse sand plains, which were now and then dotted with boulders.
Suddenly, as they ascended a long low hill, Bob and Joe cried
out in delight and pointed to something a half-mile or so away.
“A lake!” exclaimed Joe happily. “A lake of water!”
 CHAPTER XIX
Searching for the Ancient

“N OT a lake,” said Fekmah, shaking his head. “Only another

mirage. They are rather common all through this region, and we
may see much more short time.”
“Well, if there was a real lake there beside that mirage, I
wouldn’t know which to pick,” confessed Joe. “And look! Even waves
are there. And foam caps!”
“Wonderful facsimile, all right,” remarked Dr. Kirshner. “Old
Nature is capable of playing mighty big jokes on us sometimes.”
For over a half-hour the illusion was visible to the explorers;
then, when they rounded a large pile of rocks, it could no longer be
seen.
“And I’m glad,” said Bob. “Now maybe I can get my mind away
from thinking only of water. It wasn’t very pleasant to see what
looked like it and not be able to have it.”
“When do we come to another oasis?” inquired Joe of Fekmah.
“Tishmak say within next fifty miles,” was the response. “It be
very small, but there be much water to drink.”
Late that evening they came to the foothills of the Ahaggars.
Majestic Mount Oudane was directly before them, and the whole
Ahaggar range appeared to be only a few miles away in the clear
desert air.
They at last reached the small oasis among the many red
boulders. After filling their containers, they continued toward the
mountains, greatly refreshed and ready for action. But darkness was
rapidly falling, and it would be necessary to stop before long for the
night.
 Tishmak, however, thought it best to travel in the moonlight.
The others were more than willing to do this, for now that they were
so near their goal they hesitated to lose any precious time.
“We can go on for a while,” said Mr. Lewis, as daylight rapidly
faded. “Then we’ll turn in and get a few hours of sleep.”
Soon it became dark, making it necessary to stop. But before
long the moon came out in full splendor, flooding the rocky vastness
with enchanting light. The distant needle-like peaks took on a
strange appearance, like mysterious towers of a fairyland.
The scene was unusual and slightly weird, resembling the rough
surface of the moon. For some time the Americans were silent,
absorbed in thought. Finally Bob roused himself.
“Those mountains seem rather intangible, or ghost-like,” he
remarked, as he and Joe rode at the rear of the caravan.
Joe nodded.
“It’s like we’re the characters of an Arabian Nights story,” he
muttered. “No vegetation, no life of any kind around anywhere. Gets
under my skin a little.”
Through the early part of the night they rode ever on, on
toward the mysterious Ahaggars. One question stood out in the
minds of all. What did the future hold in store?
Finally Tishmak brought his dromedary to a halt beside a huge
boulder. He motioned for the others to follow suit.
“We’ll stop here for the night,” announced Fekmah, after
conversing with the guide. “But we must be up very early in morning
and get on way to mountains.”
That night everyone slept soundly, anxious to refresh
themselves thoroughly for the tiresome march through the
Ahaggars.
“Let’s go,” urged Joe, as he dressed the next morning at dawn.
“We can’t get to those hidden riches any too soon for me.”
Mr. Holton laughed unwillingly.
 “Who ever heard of fast traveling in the mountains?” he asked.
“If we make ten or fifteen miles in a day we’ll be lucky.”
“There are stretches of smooth country, though,” Dr. Kirshner
put in. “And when we get to the central plateau of the Ahaggars, it
won’t be so hard to cover territory.”
A breakfast of limited food but a bountiful supply of water was
prepared by Mr. Lewis, and then camp was broken.
In the early-morning light the peaks ahead looked pale purple,
but, said Fekmah, this color would gradually change to mauve and
blue as the sunlight became more radiant.
As they rounded a tall, red boulder, Tishmak suddenly halted his
camel and pointed to a little crevice between the rocks.
“Well, as I live!” murmured the archæologist in surprise.
“Camels—dead, mummified camels.”
The beasts had evidently been dead a long time, for their skins
were extremely dry and cracked. The fierce desert sun had
preserved their bodies for an indefinite period.
“And look, they’ve got their mouths down to the ground, as if
they were searching for water,” observed Joe.
“They were,” affirmed Fekmah. “There once a well here, but it
dried up just before camels got to it.”
“Perhaps they wandered for days searching for it, and then
finally found it—dry.” Bob shuddered.
It was a pitiable sight, particularly to the Americans. They half
expected to come across the mummified body of some unfortunate
explorer who had died a tragic death from thirst.
“We must be doubly careful to have the containers filled with
water,” warned Mr. Holton. “This is a dangerous region, and disaster
could easily come upon our little expedition.”
They trudged on in the rapidly rising temperature of the terrible
sun, keeping their eyes off the ground as much as possible to
escape the glare. They could easily have worn sun glasses, but
hesitated to do so because of the rather obstructed vision.
 “What’s this!” cried Dr. Kirshner, as they came to a huge rock
that was directly in their path.
“Some kind of an inscription, isn’t it?” inquired Bob.
“It is that!” came the excited reply. “An ancient Libyan record,
perhaps of a noteworthy event that took place in this vicinity. If you
will give me a few minutes I’ll copy this down. It may prove of great
interest in my future study of early Sahara peoples.”
The others waited for the archæologist to transcribe the writing.
It proved very difficult to read offhand, but that a full translation
would eventually come to light was not in the least doubted by the
other Americans. In fact they had come to regard Dr. Kirshner as a
wonder among men of his profession.
At last he put the paper back in its place and made a sign to
Tishmak that he was ready to continue the journey.
“Now let’s make time,” said Bob anxiously. “We ought to get
over a good many more miles before time for the noon rest.”
And they did. The country had not yet become rough enough to
hinder the progress of the dromedaries, even though huge boulders
were strewn about. By ten o’clock they had reached the base of the
Plateau of the Mouydir, a thousand-foot-high wall of solid stone.
“Tuaregs have many superstitious legends about this rock,” said
Fekmah, after talking several minutes with Tishmak. “They believe
evil spirits up in great caves come down and kill travelers. They too
think sandstorms and whirlwinds are caused by spirits hiding up in
large cracks there.”
“How interesting,” said Dr. Kirshner, getting out his small
portable typewriter.
Bob and Joe had taken motion pictures along the journey, and
now they saw another opportunity to film a scenic wonder.
“I’d like for you to do a little acting,” said Bob to Fekmah, as the
youth turned his camera in the direction of the mammoth wall of
rock.
The Arab looked up in some surprise.
 “I want you to point to the Plateau of Mouydir and talk to Dr.
Kirshner,” the young man explained. “Tell him about the legend of
the Tuaregs. Meanwhile I’ll be photographing you. Too bad this can’t
be a talking picture. All right. Let’s go.”
Fekmah understood and smiled. Dr. Kirshner was also willing to
assist the young photographers in their work.
The Arab and the American engaged in conversation, while Bob
took movies of them pointing to the high rock. When it was finished,
Bob and Joe smiled in satisfaction.
“That’s the kind of scenes we ought to have more of,” Joe said.
“They’re different from the usual monotony of ‘shooting’ the country
alone.”
“Gives a sort of individuality, huh?” laughed Mr. Holton. “Well,
any time we can be of use to you, let us know.”
Camp was made at the very base of the huge rock. Then the
usual meal was prepared.
“Use water sparingly,” cautioned Mr. Lewis, as they sat down on
the cool sand in the shade of the tent. “Tishmak says we will not
come to another well till tomorrow afternoon.”
“That’s a long time to wait,” said Dr. Kirshner gravely. “Can we
make what we have hold out?”
“We’ve got to,” Joe’s father returned. “We’ll have to restrain
from taking any undue exercise in the heat of the sun.”
“Hum-m!” Dr. Kirshner looked disappointed. “That seems to
want to spoil my plans for this afternoon.”
“How’s that?”
“I had intended to do a little exploring up on top of that wall of
stone.”
There were exclamations of surprise and anxiety.
“What!” cried Mr. Lewis. “Why, you couldn’t scale that steep cliff
with ladders and ropes!”
“Maybe not in some places,” the archæologist smiled. “But I
have noticed that there are large fissures that would offer footholds
with comparative ease, and I’m going to chance it. There’s no telling
what I may bring to light from up on that lofty rock.”
There was a period of silence, finally broken by Bob.
“May Joe and I go with you?” he asked.
There were loud protests from the youths’ fathers, who thought
it almost madness to attempt to climb the steep slope. But Dr.
Kirshner held up a hand for silence.
“Wait till we finish this meal and I’ll show you a place where it
will be more or less easy to get to the top,” he said.
“If it’s there, I’d like to see it,” came from Mr. Holton.
When the noon meal was over, the archæologist led them to a
point perhaps a quarter of a mile from the camp. He pointed up and
smiled.
“Doesn’t that look like an easy climb?” he asked. “Plenty of safe
footholds and cracks to grasp. I’m going up.”
Bob and Joe put in a request to their fathers to accompany the
scientist and were finally given permission.
“But be careful,” warned Mr. Lewis. “And don’t wander too far
away.”
Dr. Kirshner led the way up the side of the cliff, followed by Joe
and Bob. The climb was in some places difficult and a little
dangerous, but they plodded surely up.
At last, panting and perspiring, they came to the last foothold
and pulled themselves up to the top. Then they turned to take in the
view below.
Cries of astonishment came from all at the wonderful panorama
that stretched out before them. Hundreds of feet down and to their
right was the camp, and a short distance away were Mr. Holton and
Mr. Lewis. The dromedaries were tethered beside a large rock near
the cliff.
“I suppose Fekmah and Tishmak are in the tent,” remarked the
scientist, scanning the landscape.
 With the aid of his powerful binoculars the camp was made to
appear quite near, and the features of the naturalists were easily
made out.
At last Dr. Kirshner turned about.
“A fine view,” he said. “But let us not spend too much time here.
I want to explore the roof of this cliff.”
The rocky surface was in most places flat, but there were a few
huge fissures that apparently extended far into the rock.
They had come to one unusually deep crack when Dr. Kirshner
stopped and slid down the steep side, desirous of seeing the
unusual.
He reached the bottom some fifteen feet below, sending a score
of small rocks down the side of the crevice.
“What’s there?” Joe called down, bending over the side.
“Nothing, I guess. There is—— Wait a minute!”
The next moment he was all excitement, having evidently come
across something on the side of the rock.
“Drawings!” he cried animatedly, pointing to the wall about him.
“Prehistoric drawings of—of elephants!”
 CHAPTER XX
The Horror of Thirst

“E LEPHANTS?” asked Bob, almost bursting out in laughter.

“Come on,” suggested Joe, moving slowly down the side of the
fissure. “Let’s have a look at the strange drawings.”
The youths slid to the bottom, where Dr. Kirshner stood staring
at the wall.
Bob nodded.
“Drawings of elephants, all right,” he said, his eyes on the
etched rock. “And look how plain they are.”
The archæologist took out his notebook and copied the sketches
as best he could. Then he turned to the youths.
“Here is proof that the desert was not always a desert,” he said,
his eyes becoming bright with interest. “Thousands of years ago this
region was green with tropical vegetation, like the dense forests of
East Africa. It was probably inhabited by tribes of people much
different from the Arabs and Tuaregs who now live here. Then came
a gradual dry spell, and in time the luxurious growth gave way to a
hot desert of sand and rocks.”
“Those drawings of elephants, then, were made while this
region was covered with forests?” questioned Joe, becoming as
interested as the scientist.
The latter nodded.
“Elephants and other wild game probably roamed about here in
great numbers,” he explained.
After one last look at the strange sketches, the explorers began
the task of climbing up the side of the ravine. It was not easy to pull
themselves up out of the steep crevice, but the rocky walls were
solid, not even threatening to give way.
Then followed an hour of exploration about the top of the cliff,
during which time the archæologist came upon the remains of many
other ancient drawings and inscriptions. By the time that they were
ready to begin the descent of the cliff, he had filled his notebook.
“But when we get to the Ahaggars we’ll undoubtedly find many
more,” he said, slowly leading the way down.
After what seemed a long time, they came to the bottom of the
precipice and lost no time in getting back to camp.
“Have any luck?” asked Mr. Holton, looking up with interest as
the three explorers moved toward the tent.
“Did we!” laughed Joe and proceeded to tell of the many
drawings and inscriptions.
“You boys should have taken the motion-picture cameras with
you,” Mr. Lewis said. “They would have furnished proof to the
outside world.”
“Perhaps we can yet,” said Joe.
“No, you can’t,” protested Mr. Holton. “We must not waste any
time here, if we are to find the hidden riches. Right now,” he added,
“you three had better turn in and take your afternoon rest. That sun
is terrible!”
Dr. Kirshner and the youths did as suggested, glad to rest their
tired limbs. But they were up promptly at three, packing the tent
and provisions on the dromedaries.
Now, as they continued farther toward the barren mountains,
they began to realize what thirst really meant. As Tishmak had told
them, no well would be reached until late the next afternoon, and
their water containers were none too full. Their throats were
parched, and their tongues began to feel numb. The fierce sun
seemed all the hotter, greatly stimulating thirst.
All through that day they rode onward, the Ahaggars gradually
becoming nearer. It was late that night when they finally stopped
and camped in a wild region of large red rocks.
 The next day their thirst became almost overpowering, even
though they did not exercise. It seemed that they could stand it no
longer, but they rode continually on toward the well that was located
at the foot of the mountains.
The noon meal was almost without water. They did, however,
sip a small amount of the precious fluid.
“Oh, if we could only drink all we want!” groaned Joe, hesitating
to eat the beans that had been prepared. “Everything is so dry
without water.”
But although the explorers were extremely anxious to come to
the well, they gave full consideration to the midday rest. It would
have meant destruction to ride under that terrible desert sun.
“Before long we’ll come to the well,” said Fekmah, as they
prepared to continue the journey. “In an hour it be seen.”
“And how glad we’ll be,” muttered Bob, anticipating the pleasure
of drinking a large quantity of the refreshing fluid.
The hour passed slowly. They were looking about now,
searching among the many huge rocks.
Suddenly Tishmak halted abruptly, and the expression of hope
that had been on his face changed to one of fear. He motioned for
the others to move on up to where he was.
No translation of his excited words was necessary to the
Americans. They understood his anxiety. The well was dry!
For a moment the explorers sank back, and fear—stark fear—
seized them. Thoughts of disaster haunted their stricken brains—
stories of how large trading caravans had been brought to a tragic
end because of no water. It was torture unthinkable!
“And after all this waiting,” groaned Bob, his hope almost gone.
The others were equally touched. Now that they had met with
defeat, they felt at a loss to know how to carry on.
As a last resort Tishmak had fallen into a convulsion of motions
asking Allah that they might be delivered from the jaws of death. His
enthusiasm grew more intense with every moment, becoming almost
disgusting to the others. Even Fekmah, although he was a devout
believer in Mohammedanism, thought the actions of his fellow
countryman detestable.
“Come, now,” urged Dr. Kirshner, using his knowledge of the
native language to console Tishmak. “We’ll come out all right. This
isn’t the only jam we’ve been in.”
The guide finally became his natural self, although still a bit
panicky.
“You’d think after all the expeditions he’s led into the Sahara he
would be calm in the face of danger,” remarked Bob.
“Danger, yes. But not in the face of tragedy!” thought Dr.
Kirshner, although he said nothing. He feared all too much that this
might be the end.
“Where is the next well?” asked Mr. Lewis calmly.
Fekmah put the question before the guide, who replied that
there was no water within a distance of fifty miles. And mountains
lay directly before them, hindering travel. It might mean a several
days’ journey before they would come to the well, and then there
was a possibility that it, also, was dry. Disaster seemed almost
inevitable!
“But let’s hurry on,” said Mr. Holton. “Perhaps if we make time
we can get to it much sooner than we think.”
The camels were urged forward at a fast trot. But before long
they were entering the mountains, and the rapid pace was
necessarily slackened somewhat.
During that desperate ride against time, the explorers hardly
thought of the scenic wonders that lay before them. Indeed if they
had not been in such anxiety, they would have seen much to interest
them greatly.
Tall, needle-like peaks were all about, grotesque rocks dotted
the irregular plateau before them, deep gulches and ravines were
everywhere. It was a wonderful view, that beheld by the
adventurers, and could have been enjoyed to the fullest had they
not been in such terrible plight.
 Luckily there was a full moon that night, lighting the vast
expanse with a weird brightness. Countless stars shone down from
the clear sky, appearing so close that they could seemingly be
touched.
“Like we’re in another world,” breathed Bob, as he and his chum
rode rapidly at the rear of the pack camels.
“Does seem strange, doesn’t it? I wonder if we’ll live to find the
hidden riches?”
“Of course we will.” Bob cheered his friend as best he could,
and himself felt much the better for it.
Luck was with them that night. The plateau remained open and
free from peaks and rocky crags that would have delayed progress.
It was, however, very unlevel, and the dromedaries often found it
necessary to slow down to a difficult walk.
It was very late when they finally halted and made camp under
the beautiful mountain sky. After a brief supper, at which almost the
last drop of water was used, they fell asleep, not to awaken until the
sun was well up in the sky the next morning.
“You know,” remarked Fekmah, “it seems strange that that well
was dry. I been thinking about it since we left it behind. Tishmak too
thinks it strange.”
“Why?” questioned Mr. Lewis, sensing that something was in the
wind.
“Because,” Fekmah said gravely, “it a large well, and should not
go dry much easy. Tishmak think it been covered up.”
There were exclamations of surprise from the Americans.
“You mean,” began Mr. Holton, beginning to catch the point,
“that someone did it to keep us from continuing the journey?”
“Yes. I think it might have been the two thieves who stole my
map. They did it to keep us away from hidden treasure.”
There were cries of astonishment from the others. For the past
few days the thought of the thieves had been absent from their
minds. Now they began to realize that at last they had probably
come into the region in which were the hidden riches.
 “Then the rascals must be around here some place,” said Joe,
looking about sharply. “Perhaps they’re right around here.”
Fekmah got out the map he had made from memory after the
original one had been stolen. He studied it closely for a few minutes.
“Hidden treasure still great distance away,” he said at last. “We
not find it till several days pass. I think the two thieves not here but
somewhere near treasure.”
“What’s the next landmark?” inquired Mr. Lewis, as the camels
were made to move forward.
“The gorge of Arak,” Fekmah returned. “It quite a distance from
here, but Tishmak lead us to it quickly.”
All morning they trudged on without coming to the well that
Tishmak knew was somewhere in the first range of mountains.
Although it seemed impossible, their thirst rapidly increased still
more.
“Say,” cried Bob, as a sudden thought struck him, “if those two
thieves could cover up the first mountain well, they might do the
same to others. Wouldn’t it be possible?”
“Not the next one,” returned Fekmah. “It too large. Take many,
many men to stop it. But there are several small ones farther on that
could be covered.”
At an hour before noon it was necessary to stop for the daily
rest, even though they would have liked to continue in search of
water.
They were in a narrow valley between tall, sharp peaks. A
ribbon-like dry river bed wound in and out among the brightly
colored rocks, suggesting that once a rushing stream had forced its
way through the mountains.
“How I wish the river were still here,” said Joe with a sigh.
As soon as camp was made, the explorers took it easy in the
shade of the tent, more than glad to escape the terrible heat of the
sun.
But before long Bob and Joe became restless. At last Joe got up
and stretched. He sipped a very small quantity of water; then
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com