Download the Full Version of textbook for Fast Typing at textbookfull.

com

Contemporary Empirical Methods in Software

Engineering Michael Felderer

https://textbookfull.com/product/contemporary-empirical-
methods-in-software-engineering-michael-felderer/

OR CLICK BUTTON

DOWNLOAD NOW

Download More textbook Instantly Today - Get Yours Now at textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Ernst Denert Award for Software Engineering 2019: Practice

Meets Foundations Michael Felderer

https://textbookfull.com/product/ernst-denert-award-for-software-
engineering-2019-practice-meets-foundations-michael-felderer/

textboxfull.com

Effective Methods for Software Engineering First Edition

Boyd L. Summers

https://textbookfull.com/product/effective-methods-for-software-
engineering-first-edition-boyd-l-summers/

textboxfull.com

Software Engineering Perspectives in Intelligent Systems:

Proceedings of 4th Computational Methods in Systems and
Software 2020, Vol.1 Radek Silhavy
https://textbookfull.com/product/software-engineering-perspectives-in-
intelligent-systems-proceedings-of-4th-computational-methods-in-
systems-and-software-2020-vol-1-radek-silhavy/
textboxfull.com

Software Engineering Perspectives in Intelligent Systems:

Proceedings of 4th Computational Methods in Systems and
Software 2020, Vol.2 Radek Silhavy
https://textbookfull.com/product/software-engineering-perspectives-in-
intelligent-systems-proceedings-of-4th-computational-methods-in-
systems-and-software-2020-vol-2-radek-silhavy/
textboxfull.com
Managing Humans Biting and Humorous Tales of a Software
Engineering Manager Michael Lopp

https://textbookfull.com/product/managing-humans-biting-and-humorous-
tales-of-a-software-engineering-manager-michael-lopp/

textboxfull.com

Rethinking Productivity in Software Engineering Caitlin

Sadowski

https://textbookfull.com/product/rethinking-productivity-in-software-
engineering-caitlin-sadowski/

textboxfull.com

Concise Guide to Software Engineering From Fundamentals to

Application Methods 1st Edition Gerard O'Regan

https://textbookfull.com/product/concise-guide-to-software-
engineering-from-fundamentals-to-application-methods-1st-edition-
gerard-oregan/
textboxfull.com

Managing Humans Biting and Humorous Tales of a Software

Engineering Manager 3rd Edition Michael Lopp

https://textbookfull.com/product/managing-humans-biting-and-humorous-
tales-of-a-software-engineering-manager-3rd-edition-michael-lopp/

textboxfull.com

Formal Methods and Software Engineering 20th International

Conference on Formal Engineering Methods ICFEM 2018 Gold
Coast QLD Australia November 12 16 2018 Proceedings Jing
Sun
https://textbookfull.com/product/formal-methods-and-software-
engineering-20th-international-conference-on-formal-engineering-
methods-icfem-2018-gold-coast-qld-australia-
november-12-16-2018-proceedings-jing-sun/
textboxfull.com
Michael Felderer
Guilherme Horta Travassos Editors

Contemporary
Empirical Methods
in Software
Engineering
Contemporary Empirical Methods in Software
Engineering
Michael Felderer • Guilherme Horta Travassos
Editors

Contemporary
Empirical Methods in
Software Engineering
Editors
Michael Felderer Guilherme Horta Travassos
Department of Computer Science Department of Systems Engineering and
University of Innsbruck Computer Science, COPPE
Innsbruck, Austria Federal University of Rio de Janeiro
Rio de Janeiro, Brazil

ISBN 978-3-030-32488-9 ISBN 978-3-030-32489-6 (eBook)

https://doi.org/10.1007/978-3-030-32489-6

© Springer Nature Switzerland AG 2020

Chapter 17 is licensed under the terms of the Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/). For further details see licence information in the chapter.
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Foreword

As the name of the field suggests, software engineering is expected to be an engi-

neering discipline. However, it is not governed, to the same extent, by underlying
mathematical models as many other engineering disciplines, in particular, those
addressing physical artifacts as in electrical engineering or mechanical engineering.
Thus, mathematics is insufficient to conduct research and improve in software
engineering, although it is vital for some sub-areas within software engineering.
There are several reasons for this insufficiency.
First of all, the software is invisible (Brooks 1987). We can read the code, but
we cannot see it in use. We can only observe the effect of the software being
executed. Furthermore, software engineering is intrinsically complex since it is,
to a considerable extent, dependent on the knowledge and capability of humans
developing the software. Moreover, the ability of the individuals to work in a
team contributing to the same software system is essential. The development is
supported by different processes, methods, techniques, languages, and tools, which,
in one way or another, are used by the organization developing the software. Thus,
software engineering is an interplay between human knowledge, social networks
of the individuals, and available assets in the organization developing the software
(Wohlin et al. 2015).
To be able to study and improve the way software is engineered, many
researchers have embraced and promoted software engineering as an empirical
engineering discipline. Empirical studies were conducted early in the discipline,
but they were quite rare. In 1986, an article describing experimentation in software
engineering was published (Basili et al. 1986) outlining software engineering as
an experimental science. The establishment of empirical software engineering was
done to a large extent in the 1990s. At the beginning of the twenty-first century,
two books on experimentation in software engineering were published (Wohlin et
al. 2012; Juristo and Moreno 2001). The former book came in a second edition in
2012 (Wohlin et al. 2012), and it was published in Chinese in 2015.
In 2004, the concept of evidence-based software engineering was established
in software engineering (Kitchenham et al. 2004). The evidence is most often
generated from empirical studies, and hence, it was a natural continuation of the

v
vi Foreword

previous work on empirical software engineering. As the area of empirical software

engineering became well established, the need for advances in our conduct of
empirical studies grew (Shull et al. 2008). Given the applied nature of software
engineering, the need to conduct empirical studies in a real-life context was
strengthened by the publication of guidelines for conducting case studies (Runeson
et al. 2012).
As a continuation concerning the focus on evidence in software engineering, a
book on evidence-based software engineering was published in 2015 (Kitchenham
et al. 2015). Furthermore, empirical software engineering has gone from being a
sub-area of software engineering to be an integral part of software engineering.
Nowadays, it is expected that research is evaluated and assessed using empirical
methods. Thus, it is, in most cases, insufficient to present an idea or a solution
without empirical evidence. In summary, software engineering has moved into truly
being an engineering discipline.
The book Contemporary Empirical Methods in Software Engineering, edited
by Prof. Michael Felderer and Prof. Guilherme Horta Travassos, takes the next
step by including chapters on essential and timely topics in empirical software
engineering. The chapters are written by some of the world’s leading experts on
empirical methods in software engineering. The editors have done an excellent job
of attracting experts in the field who contribute with essential topics concerning the
empirical software engineering of today.
The book follows up on the previous books and articles on empirical and
evidence-based software engineering. As the title of the book suggests, the book
takes a timely step in including a set of chapters addressing emerging areas in
empirical software engineering. It provides an excellent combination of chapters
addressing contemporary areas of interest for anyone conducting research in
software engineering and in particular for those with a strong focus on empirical
software engineering. The book is a highly recommended read for, in particular,
Ph.D. students and researchers interested in conducting high-quality software
engineering research aspiring to apply empirical research methods for today and
the future.

Blekinge Institute of Technology Claes Wohlin

Karlskrona, Sweden

References

Basili VR, Selby RW, Hutchens DH (1986) Experimentation in software engineer-

ing. IEEE Trans Softw Eng SE-12(7):733–743
Brooks FP Jr (1987) No silver bullet – essence and accidents of software engineer-
ing. IEEE Comput 20(4):10–19
Juristo N, Moreno AM (2001) Basics of software engineering experimentation.
Springer, New York
Foreword vii

Kitchenham BA, Dybå T, Jørgensen M (2004) Evidence-based software engineer-

ing. In: Proceedings of 26th international conference on software engineering,
Edinburgh, pp 273–281
Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineer-
ing and systematic reviews. Chapman and Hall/CRC, Boca Raton
Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software
engineering – guidelines and examples. Wiley, Hoboken
Shull F, Singer J, Sjøberg DIK (eds) (2008) Guide to advanced empirical software
engineering. Springer, London
Wohlin C, Runeson P, Höst M, Regnell B, Ohlsson MC, Wesslén A (2012)
Experimentation in software engineering. Springer, Berlin
Wohlin C, Šmite D, Moe NB (2015) A general theory of software engineering:
balancing human, social and organizational capitals. J Syst Softw 109:229–242
Contents

The Evolution of Empirical Methods in Software Engineering .. . . . . . . . . . . . 1

Michael Felderer and Guilherme Horta Travassos

Part I Study Strategies

Guidelines for Conducting Software Engineering Research . . . . . . . . . . . . . . . . 27
Klaas-Jan Stol and Brian Fitzgerald
Guidelines for Case Survey Research in Software Engineering . . . . . . . . . . . . 63
Kai Petersen
Challenges in Survey Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 93
Stefan Wagner, Daniel Mendez, Michael Felderer, Daniel Graziotin,
and Marcos Kalinowski
The Design Science Paradigm as a Frame for Empirical Software
Engineering .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 127
Per Runeson, Emelie Engström, and Margaret-Anne Storey

Part II Data Collection, Production, and Analysis

Biometric Measurement in Software Engineering . . . . . . .. . . . . . . . . . . . . . . . . . . . 151
Fabian Fagerholm and Thomas Fritz
Empirical Software Engineering Experimentation with Human
Computation .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 173
Marta Sabou, Dietmar Winkler, and Stefan Biffl
Data Science and Empirical Software Engineering . . . . . .. . . . . . . . . . . . . . . . . . . . 217
Ezequiel Scott, Fredrik Milani, and Dietmar Pfahl
Optimization in Software Engineering: A Pragmatic Approach . . . . . . . . . . . 235
Günther Ruhe

ix
x Contents

The Role of Simulation-Based Studies in Software Engineering

Research .. . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 263
Breno Bernard Nicolau de França and Nauman Bin Ali
Bayesian Data Analysis in Empirical Software Engineering:
The Case of Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 289
Richard Torkar, Robert Feldt, and Carlo A. Furia

Part III Knowledge Acquisition and Aggregation

Automating Systematic Literature Review . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 327
Katia R. Felizardo and Jeffrey C. Carver
Rapid Reviews in Software Engineering. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 357
Bruno Cartaxo, Gustavo Pinto, and Sergio Soares
Benefitting from the Grey Literature in Software Engineering
Research .. . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 385
Vahid Garousi, Michael Felderer, Mika V. Mäntylä, and Austen Rainer
Guidelines for Managing Threats to Validity of Secondary Studies
in Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 415
Apostolos Ampatzoglou, Stamatia Bibi, Paris Avgeriou,
and Alexander Chatzigeorgiou
Research Synthesis in Software Engineering . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 443
Paulo Sérgio Medeiros dos Santos and Guilherme Horta Travassos

Part IV Knowledge Transfer

Open Science in Software Engineering . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 477
Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold
Third Generation Industrial Co-production in Software Engineering. . . . . 503
Tony Gorschek and Krzysztof Wnuk
The Evolution of Empirical Methods
in Software Engineering

Michael Felderer and Guilherme Horta Travassos

Abstract Empirical methods like experimentation have become a powerful means

to drive the field of software engineering by creating scientific evidence on software
development, operation, and maintenance, but also by supporting practitioners in
their decision-making and learning. Today empirical methods are fully applied in
software engineering. However, they have developed in several iterations since the
1960s. In this chapter we tell the history of empirical software engineering and
present the evolution of empirical methods in software engineering in five iterations,
i.e., (1) mid-1960s to mid-1970s, (2) mid-1970s to mid-1980s, (3) mid-1980s to end
of the 1990s, (4) the 2000s, and (5) the 2010s. We present the five iterations of
the development of empirical software engineering mainly from a methodological
perspective and additionally take key papers, venues, and books, which are covered
in chronological order in a separate section on recommended further readings, into
account. We complement our presentation of the evolution of empirical software
engineering by presenting the current situation and an outlook in Sect. 4 and the
available books on empirical software engineering. Furthermore, based on the
chapters covered in this book we discuss trends on contemporary empirical methods
in software engineering related to the plurality of research methods, human factors,
data collection and processing, aggregation and synthesis of evidence, and impact
of software engineering research.

Guilherme Horta Travassos is a CNPq Researcher.

M. Felderer ()
Department of Computer Science, University of Innsbruck, Innsbruck, Austria
Department of Software Engineering, Blekinge Institute of Technology, Karlskrona, Sweden
e-mail: [email protected]
G. H. Travassos
Department of Systems Engineering and Computer Science, COPPE, Federal University of Rio
de Janeiro, Rio de Janeiro, Brazil
e-mail: [email protected]

© Springer Nature Switzerland AG 2020 1

M. Felderer, G. H. Travassos (eds.), Contemporary Empirical Methods in Software
Engineering, https://doi.org/10.1007/978-3-030-32489-6_1
2 M. Felderer and G. H. Travassos

1 Introduction

The term software engineering originated in the early 1960s (Hey et al. 2014).
During the NATO Software Engineering Conferences held in 1968 and 1969,
participants made explicit that engineering software requires dedicated approaches
that are separate from those for the underlying hardware systems. Until that
“software crisis,” software-related research mostly focused on theoretical aspects,
e.g., algorithms and data structures used to write software systems, or practical
aspects, e.g., an efficient compilation of software for particular hardware sys-
tems (Guéhéneuc and Khomh 2019). Since then, these topics are investigated in
computer science, which pertains to understanding and proposing theories and
methods related to the efficient computation of algorithms, and differs from software
engineering (research), which has become a very dynamic discipline on its own
since its foundation in the 1960s. IEEE (1990, 2010) defines software engineering
(SE) as: (1) The application of a systematic, disciplined, quantifiable approach to
the development, operation, and maintenance of software, that is, the application
of engineering to software, and (2) The study of approaches as in (1). Software
engineering also differs from other engineering disciplines due to the immaterial
nature of software not obeying physical laws and the importance of human factors
as software is written by people for people. Software engineering is fundamentally
an empirical discipline, where knowledge is gained applying direct and indirect
observation or experience. Approaches to software development, operation, and
maintenance must be investigated by empirical means to be understood, evaluated,
and deployed in proper contexts. Empirical methods like experimentation are
therefore essential in software engineering to gain scientific evidence on software
development, operation, and maintenance, but also to support practitioners in their
decision-making and learning (Travassos et al. 2008). The application of empirical
methods makes software engineering more objective and less imprecise, facilitating
the transfer of software technologies to the industry (Shull et al. 2001). Software
engineers learn by observing, exploring, and experimenting. The level of learning
depends on the degree of observation or intervention (Thomke 2003) promoted by
the experiences and studies performed.
Traditionally, empirical software engineering (ESE) is the area of research that
emphasizes the use of empirical methods in the field of software engineering.
According to Harrison and Basili (1996), “Empirical software engineering is the
study of software-related artifacts for the characterization, understanding, eval-
uation, prediction, control, management, or improvement through qualitative or
quantitative analysis. The quantitative studies may range from controlled experi-
mentation to case studies. Qualitative studies should be well-defined and rigorous.”
The role and importance of the different types of empirical methods in software
engineering have evolved since the foundation of software engineering. In this
chapter, we discuss the evolution of empirical methods in software engineering and
especially also take key venues and books into account as they reflect that evolution.
The Evolution of Empirical Methods in Software Engineering 3

The chapter is organized as follows: In Sect. 2, we provide background on

empirical research methods in software engineering. In Sect. 3, we present the
evolution of empirical software engineering by describing five iterations of its devel-
opment. Based on that “historical” perspective on empirical software engineering,
in Sect. 4 we describe current trends in empirical software engineering based on the
chapters on contemporary empirical methods in software engineering covered in this
book. In Sect. 5, we present the available books on empirical methods in software
engineering in chronological order as recommended further reading. Finally, in
Sect. 6, we conclude this chapter.

2 Empirical Research Methods in Software Engineering

The scientific approach typically consists of observation, measurement, and exper-

imentation. Observation helps researchers to formulate essential questions about
a phenomenon under study to build models and to derive hypotheses that can
be tested through experimentation. Measurement is essential for both observation
and experimentation. A scientific hypothesis must be refutable to be meaningfully
tested. Tested hypotheses are compiled and communicated in the form of laws or
theories. At the heart of the scientific approach are research methods in general and
the empirical method in particular. Empirical methods leverage evidence obtained
through observation, measurement, or experimentation to address a scientific
problem. Evidence should be based on qualitative and quantitative research. In this
section, we provide an overview of research methods in software engineering in
general and empirical methods in particular.

2.1 Research Methods

To perform scientific research in software engineering, one has to understand

the available research methods and their limitations. For the field of software
engineering, Basili (1993) and Glass (1994) summarized four research methods:
scientific, engineering, empirical, and analytical.
The so-called scientific method observes the world and builds a model based on
the observations, e.g., a simulation model of the software process or product. The
scientific method is inductive and tries to extract from the world some model that
can explain a phenomenon and to evaluate whether the model is representative for
the phenomenon under observation. It is a model-building approach.
The engineering method studies current solutions, proposes changes, and then
evaluates them. It suggests the most appropriate solutions, develops, measures and
analyzes, and repeats until no further improvement is possible. It is an evolutionary
improvement-oriented approach that assumes the existence of some model of the
software process or product. It modifies this model to improve the objects of study.
4 M. Felderer and G. H. Travassos

The empirical method proposes a model and evaluates it through empirical

studies like case studies or experiments. The empirical method normally follows
an iterative and incremental approach that can begin with an exploratory survey,
followed by case studies in an industrial context to better understand specific
phenomena and controlled experiments to investigate cause–effect relationships.
The analytical method proposes a formal theory, develops the theory, derives the
results, and, if possible, compares it with empirical observations. It is deductive and
provides an analytical basis for developing a model.
Traditionally, the analytical method is used in the more formal areas of electrical
engineering and computer science, but is important for software engineering as
well, e.g., when building mathematical models for software reliability growth (Lyu
et al. 1996). The scientific method, inspired by natural science, is traditionally
used in applied areas, such as the simulation of a sensors network to evaluate
its performance. However, simulations are used as a means for conducting an
experiment as well (Baros et al. 2004). The engineering method is dominating
in industry (Wohlin et al. 2012). The empirical method, mainly using empirical
strategies, has traditionally been used in social sciences and psychology, where
one is unable to state any laws of nature but concerned with human behavior. The
engineering and the empirical method can be seen as variations of the scientific
method (Basili 1993). This overlap and an integrated view of the scientific, engineer-
ing, and empirical methods is also an underlying design principle of this book on
empirical methods. It considers not only chapters on traditional empirical strategies
like surveys (see chapter “Challenges in Survey Research”), but for instance, also a
chapter on simulation-based studies (see chapter “The Role of Simulation-Based
Studies in Software Engineering Research”), which are closer to the scientific
method as defined above, or a chapter on design science (see chapter “The Design
Science Paradigm as a Frame for Empirical Software Engineering”), which can
tightly be linked to the engineering method. All of these investigation strategies
refer to empirical methods.

2.2 Empirical Methods

Empirical methods rely on the collected data. Data collection methods may involve
qualitative or quantitative data. Some widely used qualitative data collection
methods in software engineering are interviews and participant observation (Seaman
1999). Some commonly used quantitative data collection methods are archival data,
surveys, experiments, and simulation (Wohlin et al. 2012). Once data are collected,
the researcher needs to analyze the data by using qualitative analysis methods,
e.g., grounded theory, thematic analysis, or hermeneutics, and quantitative analysis
methods, e.g., statistical analysis and mathematical modeling approaches.
In general, there are three widely recognized research processes called quanti-
tative research, qualitative research, and semiquantitative research. An alternative
option is the combination of both qualitative and quantitative research, denoted as
The Evolution of Empirical Methods in Software Engineering 5

mixed research (Creswell and Creswell 2018). The distinction between qualitative
and quantitative research comes not only from the type of data collected, but also
the objectives, types of research questions posed, analysis methods, and the degree
of flexibility built into the research design as well (Wohlin and Aurum 2015).
Qualitative research aims to understand the reason (i.e., “why”) and mechanisms
(i.e., “how”) explaining a phenomenon. A popular method of qualitative research
is case study research, which examines a set of selected samples in detail to
understand the phenomenon illustrated by the samples. For instance, a qualitative
study can be conducted to understand the impediments of automating system
tests. Quantitative research is a data-driven approach used to gain insights about
an observable phenomenon. Data collected from observations are analyzed using
mathematical and statistical models to derive quantitative relationships between
different variables capturing different aspects of the phenomenon under study. A
popular method of quantitative research are controlled experiments to examine
cause–effect relationships between different variables characterizing a phenomenon
in a controlled environment. For instance, different review techniques could be
compared via a controlled experiment. Mixed research collects quantitative and
qualitative data. It is a particular form of multi-method research, which combines
different research methods to answer some hypotheses, and is often used in empir-
ical software engineering due to the lack of theories in software engineering with
which we interpret quantitative data and due to the need to discuss qualitatively the
impact of the human factor on any experiments in software engineering (Guéhéneuc
and Khomh 2019). Semiquantitative research deals with approximate measurements
to data rather than exact measurements (Bertin 1978). It looks for understanding the
behavior of a system based on causal relations between the variables describing
the system. Semiquantitative models allow one to express what is known without
making inappropriate assumptions, simulating ranges of behavior rather than values
of point (Widman 1989). It has many applications in both the natural and social
sciences. Semiquantitative research supports cases where direct measurements are
not possible, but where it is possible to estimate an approximated behavior. In other
words, this type of study is applied in scenarios where the numerical values in the
mathematical relations governing the changes of a system are not known. In this
context, the direction of change is known, but not the size of its effect (Ogborn and
Miller 1994). Simulation-based studies in software engineering can benefit from
using semiquantitative research (Araújo et al. 2012).
The three major and well-established empirical methods in software engineering
are: survey, case study, and experiment (Wohlin et al. 2012). Primary studies using
such methods can be performed in vivo, in vitro, in virtuo, and in silico (Travas-
sos and Barros 2003). In vivo studies involve participants and projects in their
natural environments and contexts. Such studies are usually executed in software
development organizations throughout the software development process under real
working conditions. In vitro studies are performed in controlled environments, such
as laboratories or controlled communities, under configured working conditions. In
virtuo studies have the subjects interacting with a computerized model of reality.
The behavior of the environment with which subjects interact is described as a
6 M. Felderer and G. H. Travassos

model and represented by a computer program. In silico studies represent both

subjects and real world as computer models. The environment is fully composed
of computer models to which human interaction is reduced to a minimum.
A survey is a system for collecting information from or about subjects (people,
projects, among others) to describe, compare, or explain their knowledge, attitudes,
and behavior (Fink 2003). A survey is often an investigation performed in retrospect,
when, for instance, a tool or technique has been in use for a while (Pfleeger 1995).
The primary means of gathering qualitative or quantitative data are interviews or
questionnaires. These are done through taking a sample that is representative of the
population to which is generalized.
A case study in software engineering is an empirical inquiry that draws on
multiple sources of evidence to investigate one or a small number of instances
of a contemporary software engineering phenomenon within its real-life context,
especially when the boundary between phenomenon and context cannot be clearly
specified (Runeson et al. 2012).
An experiment is used to examine cause–effect relationships between different
variables characterizing a phenomenon (Guéhéneuc and Khomh 2019). Experi-
ments allow researchers to verify, refute, or validate hypotheses formulated about
the phenomenon under study. In a controlled experiment, one variable of the study
setting is manipulated, and based on randomization, different treatments are applied
to or by different subjects while keeping other variables constant, and measuring
the effects on outcome variables (Wohlin et al. 2012). A quasi-experiment is similar
to a controlled experiment, where the assignment of treatments to subjects cannot
be based on randomization, but emerges from the characteristics of the subjects
or objects themselves (Wohlin et al. 2012). Replication experiments reproduce
or quasi-reproduce previous experiments with the objectives to confirm or infirm
the results from previous experiments or to contrast previous results in different
contexts (Guéhéneuc and Khomh 2019).
Regardless of the applied empirical method, to acquire scientific evidence about
the investigated software engineering phenomena involves observation, measure-
ment, and experimentation of the world and existing solutions. It demands the
proposition of models and theories describing the observed behavior, collecting
and analyzing data, putting the hypotheses under proof, and repeating the overall
process over time to strengthen the evidence on the observed phenomena. Based on
several primary studies, in which direct observations and measurements about the
objects of interest are made, whether by surveys, experiments, or case studies, which
are there also called empirical strategies, one can perform secondary studies. A
secondary study does not generate any data from direct observation or measurement,
instead, it analyzes a set of primary studies and usually seeks to aggregate the
results from these to provide stronger forms of evidence about a particular phe-
nomenon (Kitchenham et al. 2015). Secondary studies typically appear as systematic
(literature) reviews, which aim to provide an objective and unbiased approach to
finding relevant primary studies, and for extracting, aggregating, and synthesizing
the data from these (Kitchenham et al. 2015). A particular type of a systematic
review is a systematic mapping study (Petersen et al. 2015), which classifies studies
The Evolution of Empirical Methods in Software Engineering 7

to identify clusters of studies (that could form the basis of a fuller review with more
synthesis) and gaps indicating the need for more primary studies.
The scientific or industrial significance of empirical studies depends on their
validity, i.e., the degree to which one can trust the outcomes of an empirical
study (Kitchenham et al. 2015). Validity is usually assessed in terms of four
commonly encountered forms of threats to validity: internal, external, construct,
and conclusion validity (Shadish et al. 2002). Internal validity refers to inferences
that the observed relationship between treatment and outcome reflects a cause–
effect relationship. External validity refers to whether a cause–effect relationship
holds over other conditions, including persons, settings, treatment variables, and
measurement variables. Construct validity refers to how concepts are operational-
ized as experimental measures. Conclusion validity refers to inferences about the
relationship between treatment and outcome variables.
The accomplishment of empirical studies relies on performing well-defined and
evolutionary activities. The classical empirical study process consists of five phases:
definition, planning, operation, analysis, and interpretation, as well as reporting and
packaging (Juristo and Moreno 2001; Malhotra 2016). The definition phase makes
the investigated problem and overall objectives of the study explicit. The planning
phase covers the study design and includes the definition of research questions and
hypotheses as well as the definition of data collection, data analysis, and validity
procedures. In the operation phase, the study is actually conducted. In the analysis
and interpretation phase, the collected data is analyzed, assessed, and discussed.
Finally, in the reporting and packaging phase, the results of the study are reported
(e.g., in a journal article, a conference paper, or a technical report) and suitably
packaged to provide study material and data. The latter has become more critical
recently due to the open science movement (see chapter “Open Science in Software
Engineering”).

3 Evolution of Empirical Software Engineering

The application of empirical methods in general and empirical software engineering

in particular is well-established in software engineering research. Almost all papers
published in major software engineering venues these days include an empirical
study (Theisen et al. 2017). Furthermore, since 2000, research methodology has
received considerable attention in the software engineering research community
resulting in many available publications on empirical research methodology in
software engineering. In a recent mapping study, Molléri et al. (2019) identified
341 methodological papers on empirical research in software engineering.
The application of empirical methods and the underlying research methodology
has developed iteratively since the foundation of software engineering in the 1960s.
Guéhéneuc and Khomh (2019) discuss landmark articles, books, and venues in
empirical software engineering that indicate the iterative development of the field.
Bird et al. (2015) distinguish four “generations” of analyzing software data, i.e.,
8 M. Felderer and G. H. Travassos

preliminary work, academic experiments, industrial experiments, and “data science

everywhere.” In this section, we present five iterations of the development of
empirical software engineering from a methodological perspective. We additionally
take articles and venues into account, which is needed for a holistic understanding
of the field’s development. We complement our presentation of the evolution of
empirical software engineering by presenting the current situation and an outlook in
Sect. 4 and the available books on empirical software engineering in chronological
order in Sect. 5 on recommended further reading.

3.1 First Iteration: Mid-1960s to Mid-1970s

In the early years of software engineering, empirical studies were rare, and the
only research model commonly in use was the analytical method, where different
formal theories were advocated devoid of any empirical evaluation (Glass 1994).
According to a systematic literature review of empirical studies performed by
Zendler (2001), Grant and Sackman (1967) published the first empirical study in
software engineering in 1967. The authors conducted an experiment that compared
the performance of two groups of developers, one working with online access to
a computer through a terminal and the other with offline access in batch mode.
Another empirical study published early in the history of software engineering was
an article by Knuth (1971), in which the author studied a set of Fortran programs
to understand what developers do in Fortran programs. Akiyama (1971) describes
the first known “size law” (Bird et al. 2015), stating that the number of defects is a
function of the number of lines of code. The authors in these and other early studies
defined the goal of the study, the questions to research, and the measures to answer
these questions in an ad hoc fashion (Guéhéneuc and Khomh 2019). However, they
were pioneers in the application of empirical methods in software engineering.

3.2 Second Iteration: Mid-1970s to Mid-1980s

In the second iteration, already more empirical studies, mainly in vitro experiments,
were conducted. Prominent examples are experiments on structured program-
ming (Lucas et al. 1976), flowcharting (Shneiderman et al. 1977), and software
testing (Myers 1978). The second iteration is characterized by first attempts to pro-
vide a systematic methodology to define empirical studies in software engineering in
general and experiments in particular. These attempts culminated in the definition of
the Goal/Question/Metrics (GQM) approach by Basili and Weiss (1984). The GQM
approach helped practitioners and researchers to define measurement programs
based on goals related to products, processes, and resources that can be achieved
by answering questions that characterize the objects of measurement using metrics.
The Evolution of Empirical Methods in Software Engineering 9

The methodology has been used to define experiments in software engineering

systematically.
In that iteration, empirical software engineering was also institutionalized for
the first time. In 1976, the NASA Goddard Software Engineering Laboratory
(NASA/SEL) was established at the University of Maryland, College Park (USA),
aiming to support the observation and understanding of software projects (Basili
and Zelkowitz 2007). The establishment of NASA/SEL provided the means to
strengthen the importance of using basic scientific and engineering concepts in
the context of software engineering (McGarry et al. 1994). The paradigm change
provided by using GQM (Basili and Weiss 1984), including the ability of packaging
knowledge on how to better build a software system, improved the way experiences
could be organized and shared. The building and evolution of models at NASA/SEL
pave the road for organizing the Experience Factory model (Basili et al. 1994) and
the dissemination of initial good practices on empirical software engineering.

3.3 Third Iteration: Mid-1980s to End of the 1990s

In the third iteration, not only experiments but also surveys (for instance,
by Burkhard and Jenster (1989) on the application of computer-aided software
engineering tools) and case studies (for instance, by Curtis et al. (1988) on the
software design process for large systems) were performed to some extent. Also,
the explicit discussion of threats to validity appeared in that iteration. One of the first
studies explicitly discussing its threats to validity was an article by Swanson and
Beath (1988) on the use of case study data in software management research. From
the late 1980s, researchers also started to analyze software data using algorithms
taken from artificial intelligence research (Bird et al. 2015). For instance, decision
trees and neural networks were applied to predict error-proneness (Porter and
Selby 1990), to estimate software effort (Srinivasan and Fisher 1995) and to model
reliability growth (Tian 1995).
In the third iteration, empirical studies began to attract the attention of several
research groups all over the world, who realized the importance of providing empir-
ical evidence about the developed and investigated software products and processes.
The experiences shared by NASA/SEL and the participation of several researchers
in conducting experiments together with NASA/SEL helped to strengthen the use
of different experimental strategies and the application of surveys.
The interest in the application of the scientific method by different researchers,
the identification of the need to evolve the experimentation process through sharing
of experimental knowledge among peers as well as the transfer of knowledge to
industry, among other reasons, led to the establishment of the International Software
Engineering Research Network (ISERN) in 1992. ISERN held its first annual
meeting in Japan in 1993 sponsored by the Graduate School of Information Science
at the Nara Institute of Science and Technology.
10 M. Felderer and G. H. Travassos

The need to share the ever increasing number of studies and their results and the
growing number of researchers applying empirical methods in software engineering
lead to the foundation of suitable forums. In 1993 the IEEE International Software
Metrics Symposium, in 1996, the Empirical Software Engineering International
Journal, and in 1997, the Empirical Assessments in Software Engineering (EASE)
event at Keele University were founded.
By the end of this iteration, several institutes dedicated to empirical software
engineering were established. In 1996, the Fraunhofer Institute for Experimental
Software Engineering (IESE) associated with the University of Kaiserslautern
(Germany) was established. In 1998, the Fraunhofer Center for Experimental
Software Engineering (CESE) associated with the University of Maryland, College
Park (USA) began operations. Also, other institutions and laboratories, such as
National ICT Australia as well as the Simula Research Laboratory and SINTEF
(both located in Norway), among others, started to promote empirical studies in
software engineering in the industry.
Finally, by the end of the 1990s, the publication of methodological papers on
empirical methods in software engineering started. Zelkowitz and Wallace (1998)
provided an overview of experimental techniques to validate new technologies,
Seaman (1999) provided guidelines for qualitative data collection and analysis, and
Basili et al. (1999) discussed families of experiments.

3.4 Fourth Iteration: The 2000s

Since 2000 research methodology has received considerable attention, and therefore
the publication of methodological papers further increased. For instance, Höst et al.
(2000) discuss the usage of students as subjects in experiments, Shull et al. (2001)
describe a methodology to introduce software processes based on experimentation,
Pfleeger and Kitchenham (2001) provide guidelines on surveys in software engi-
neering, Lethbridge et al. (2005) provide a classification of data collection methods,
Kitchenham and Charters (2007) provide guidelines for performing systematic
literature reviews in software engineering, Shull et al. (2008) discuss the role
of replication in empirical software engineering, and Runeson and Höst (2009)
provide guidelines for case study research. In connection to the increased interest
in research methodology, also the first books on empirical research methods in
software engineering with a focus on experimentation written by Wohlin et al.
(2000) and Juristo and Moreno (2001) appeared around 2000 (see Sect. 5 for
a comprehensive overview of books on empirical software engineering). Also,
combining research methods and performing multi-method research became more
popular in the period. One of the first papers following a multi-method research
methodology was published by Espinosa et al. (2002) on shared mental models,
familiarity, and coordination in distributed software teams.
With the growing number of empirical studies, knowledge aggregation based
on these primary studies became more crucial to understand software engineering
The Evolution of Empirical Methods in Software Engineering 11

phenomena better. No single empirical study on a software engineering phe-

nomenon can be considered definitive (Shull et al. 2004) and generalized to any
context. Therefore, the replication of studies in different contexts is of paramount
importance to strengthen its findings. However, the existence of conclusive, no
conclusive, contradictory, and confirmatory results about a particular software
engineering phenomenon should be combined to strengthen the evidence on the
software phenomena or to reveal the need for more primary studies on phenomenon
of interest. In consequence, there arose a need for secondary studies that aim
to organize, aggregate, and synthesize all relevant results from primary studies
regarding a particular phenomenon under research. Kitchenham (2004) was the
first who recommended the use of systematic literature reviews (SLRs) in software
engineering and adapted respective guidelines, mainly from medical research, to
software engineering. With the guidelines of Kitchenham (2004) and Biolchini
(2005), the empirical software engineering community had a tool to systemat-
ically synthesize knowledge available in primary studies, which spread rapidly
and enabled evidence-based software engineering (Kitchenham et al. 2004). In
a systematic review of SLRs in software engineering, Kitchenham and Brereton
(2013) identified 68 papers reporting 63 unique SLRs published in SE conferences
and journals between 2005 and mid-2012. Petersen et al. (2008) clarify and expand
upon the differences between SLRs and systematic mapping studies and provide
guidelines for the latter. In their seminal paper on the future of empirical methods
in software engineering research, Sjøberg et al. (2007) present the important role
of synthesis of empirical evidence in their vision of software engineering research.
The vision is that for all fields of software engineering, empirical research methods
should enable the development of scientific knowledge about how useful different
SE technologies are for different kinds of actors, performing different kinds of
activities, on different kinds of systems to guide the development of new SE
technology and important SE decisions in industry. Major challenges to the pursuit
of this vision are more and better synthesis of empirical evidence, and connected
to that building and testing more theories as well as increasing quality, including
relevance, of studies.
One of the problems faced by the software engineering community has often
been the scarcity of software data for conducting empirical studies (Malhotra 2016).
The availability of (open) source code repositories and software process data due
to automated or even continuous software engineering enabled new data mining
approaches in software engineering in that period. In a seminal paper, Zimmermann
et al. (2005) used association rule learning to find patterns of defects in a large
set of open-source projects. Furthermore, also, software data from companies were
analyzed. For instance, at AT&T, Ostrand et al. (2004) used code metrics to
predict defects, and at Microsoft—which even founded an own Empirical Software
Engineering (ESE) group in Microsoft Research (Bird et al. 2011)—Nagappan and
Ball (2005) showed that data from that organization could predict software quality.
In consequence, also repositories—like the PROMISE repository—that collect
software data and make them publicly available were founded. The PROMISE
repository was founded in 2005 and seeded with NASA data (Menzies et al. 2014).
12 M. Felderer and G. H. Travassos

The empirical evidence gathered through analyzing the data collected from the
software repositories is considered to be an important support for the (empirical)
software engineering community these days. There are even venues that focus on
analysis of software data such as Mining Software Repositories (MSR), which was
organized for the first time in 2004 in Edinburgh (UK) and Predictive Models and
Data Analytics in Software Engineering (PROMISE), which was organized for the
first time in 2005 in St. Louis (USA).
In general, the growing interest in empirical software engineering in that period
resulted in projects such as the Experimental Software Engineering Research
Network (ESERNET) in Europe from 2001 to 2003 and the foundation of several
venues. In 2007, the first ACM/IEEE International Symposium on Empirical
Software Engineering and Measurement (ESEM) was held in Madrid (Spain).
ESEM is the result of the merger between the ACM/IEEE International Symposium
on Empirical Software Engineering, which ran from 2002 to 2006, and the IEEE
International Software Metrics Symposium, which ran from 1993 to 2005. In 2003,
Experimental Software Engineering Latin American Workshop (ESELAW) was
organized for the first time. Also, in 2003, the International Advanced School
on Empirical Software Engineering (IASESE) performed its first set of classes in
Rome (Italy). In 2006, the International Doctoral Symposium on Empirical Software
Engineering (IDoESE) was founded. Today, the ISERN annual meeting, IASESE,
IDoESE, and ESEM form the Empirical Software Engineering International Week
(ESEIW), which is held annually.

3.5 Fifth Iteration: The 2010s

Since 2010 empirical studies are “everywhere” in software engineering. Almost

all papers in major software engineering conferences like ICSE contain empirical
studies. Also, more and more books dedicated to empirical research methodology
in software engineering are published (see Sect. 5), and papers on empirical research
methodology are published at a constant pace. For instance, Ivarsson and Gorschek
(2011) present a model for evaluating the rigor and relevance of technology
evaluations in industry, Arcuri and Briand (2014) provide a guide to statistical tests
for assessing randomized algorithms in software engineering, Wieringa (2014a)
discusses scaling up of empirical methods for technology validation in practice,
Wohlin and Aurum (2015) provide a decision-making structure for selecting a
research design, de Mello et al. (2015) provide probabilistic sampling approaches
for large-scale surveys, Sharp et al. (2016) discuss the use and value of ethnographic
studies in software engineering research, Stol et al. (2016) discuss the use of
grounded theory and their reporting, Briand et al. (2017) discuss the importance
of context and the overrating of generalizability in software engineering, and
Stol and Fitzgerald (2018) provide a holistic framework for software engineering
research. Furthermore, Harman et al. (2010) provide a comprehensive overview and
guidance on the application of search-based optimization in software engineering.
The Evolution of Empirical Methods in Software Engineering 13

Especially, in this period many papers presenting results on search-based software

engineering, that generally (though not exclusively) fall in the category of empirical
software engineering papers were published. Due to the potentially high compu-
tational complexity of optimization algorithms, some researchers have started to
use high performance computing environments to support the execution of their
studies (Farzat et al. 2019).
In this iteration, one can observe a growing interest in the role of theory within
software engineering research to develop the field further as a scientific discipline. In
December 2009, the Software Engineering Method and Theory (SEMAT) initiative
was launched that aims towards the development of a general theory of software
engineering. SEMAT organized several events, among others, a workshop series
on a General Theory of Software Engineering (GTSE) between 2012 and 2015.
Stol and Fitzgerald (2015) even argue for a theory-oriented software engineering
research perspective, which can complement the recent focus on evidence-based
software engineering. Also, several concrete theories have been developed in that
iteration. For instance, Johnson and Ekstedt (2016) present a general theory of
software engineering called Tarpit, Bjarnason et al. (2016) a theory of distances
in software engineering, and Wagner et al. (2019) a theory on requirements
engineering.
Today not only almost all papers in major software engineering conferences
contain empirical studies, but also most software engineering conferences have
explicitly integrated empirical software engineering into their program, e.g., as
dedicated sessions or tracks. In addition, there are several workshops on conducting
empirical studies in specific areas. For instance, at ICSE, there has been a collocated
International Workshop on Conducting Empirical Studies in Industry (CESI) and at
RE the International Workshop on Empirical Requirements Engineering (EmpiRE).
The Experimental Software Engineering Latin American Workshop (ESELAW)
joined the Ibero-American Conference on Software Engineering (CIbSE) in 2011
as a colocated workshop and became a dedicated track in 2013 due to the increased
number of submissions.
ESEIW, including ESEM, and EASE are established as the two leading annual
events to discuss methodological issues on empirical research in software engi-
neering. Empirical methods have been an explicit topic in several summer schools
including the annual LASER summer school (which hosted the topic empirical
software engineering in 2010), PASED—Canadian Summer School on Practical
Analyses of Software Engineering Data in 2011, the Empirical Research Meth-
ods in Software Engineering and Informatics (ERMSEI) in 2016 and 2017, the
International Summer School on Software Engineering (SIESTA) in 2018 and 2019
as well as the 2019 Summer School in Empirical Software Engineering at Brunel
(UK). In the context of ESEIW, the International Advanced School on Empirical
Software Engineering (IASESE) has been organized annually since 2003 and helped
to spread knowledge on current empirical methods in software engineering among
junior and senior researchers. Figure 1 presents the IASESE timeline and its topics
along the places and years. The topics taught over the years also reflect the evolution
of empirical software engineering, as discussed in this section.
 14

2003 – Rome, Italy 2019 – Porto de Galinhas, Brazil

Evaluating the maturing of a technology for use Observation as a Data Collection Technique
Using Empirical study to do technology transfer for Software Engineering Research
Using qualitative analysis in software engineering
Building parametric models 2018 – Oulo, Finland
Empirical Software Engineering and Data
2004 – Los Angeles, USA
Science: Old Wine in a New Bottle
How to do Empirical Research IASESE
2005 – Noosa Heads, Australia International Advanced School on 2017 – Toronto, Canada
Research Protocols and Systematic Empirical Software Engineering Product Development and Innovation with
Literature Reviews Continuous Experimentation
2006 – Rio de Janeiro, Brazil 2016 – Ciudad Real, Spain
Technology Evaluation Locations and Topics
Surveys in Software Engineering
How to run empirical studies using project 2003 - 2019
repositories (and avoid common pitfalls) 2015 – Beijing, China
The role of replication in Software Engineering Theories in Empirical Software Engineering
Software engineering process simulation
2007 – Madrid, Spain 2014 – Torino, Italy
Case Study Research In Vivo Experimentation in Software
Engineering
2008 – Kaiserslautern, Germany
Replication and Aggregation of Software 2013 – Baltimore, USA
Engineering Experiments Action Research
2009 – Orlando, USA
Selecting Research Methods for Empirical 2012 – Lund, Sweden
Software Engineering Evidence Synthesis of Qualitative Research

2010 – Bolzano-Bozen, Italy 2011 – Banff, Canada

Using Ethnographic Methods in Empirical Evidence-based Decision-Support in
Software Engineering Research Software Engineering

Fig. 1 International Advanced School on Empirical Software Engineering (IASESE) timeline and topics from 2003 to 2019
M. Felderer and G. H. Travassos
The Evolution of Empirical Methods in Software Engineering 15

4 Current Situation and Outlook

Since the first empirical studies in the 1960s, the field of empirical software
engineering has considerably matured in several iterations. However, the empirical
methods resulting from the five iterations presented in the previous section are
not the end of the story, and as in any scientific discipline, research methods
develop further. The chapters of this book discuss contemporary empirical methods
that impact the current evolution of empirical software engineering and form the
backbone of its next iteration. For sure, the description of the current situation
and future trends is never complete and always subjective to some extent. But
we think that the chapters covered in this book show several interesting trends
in contemporary empirical methods in software engineering, which we want to
summarize here.
The evolution of empirical software engineering leads to the continuous adoption
of empirical methods from other fields and the refinement of existing empirical
methods in software engineering. The resulting plurality of research methods
requires guidance in knowledge-seeking and solution-seeking (i.e., design science)
research. The chapter “Guidelines for Conducting Software Engineering Research”
presents guidelines for conducting software engineering research based on the
ABC framework, where ABC represents the three desirable aspects of research
generalizability over actors (A), precise control of behavior (B), and realism of
context (C). Each empirical method has its strengths and weaknesses. It is beneficial
to utilize a mix of methods depending on the research goal or even to combine
methods. Case survey research combines case study and survey research, which rely
primarily on qualitative and quantitative data, respectively. The chapter “Guidelines
for Case Survey Research in Software Engineering” provides an overview of the
case survey method. While being an important and often used empirical method,
survey research has been less discussed on a methodological level than other types
of empirical methods. The chapter “Challenges in Survey Research” discusses
methodological issues in survey research for software engineering concerning
theory building, sampling, invitation and follow-up, statistical analysis, qualitative
analysis, and assessment of psychological constructs. Although software engineer-
ing is an engineering discipline, the design science paradigm has been explicitly
adapted to software engineering relatively late by Wieringa (2014b), and the full
potential of the design science paradigm has not been exploited so far in software
engineering. The chapter “The Design Science Paradigm as a Frame for Empirical
Software Engineering” uses the design science paradigm as a frame for empirical
software engineering and uses it to support the assessment of research contributions,
industry-academia communication, and theoretical knowledge building.
It is generally acknowledged that software development is a human-intensive
activity as software is built by humans for humans. However, traditionally SE
research has focused on artifacts and processes without explicitly taking human
factors in general and the developer perspective in particular into account. If the
perspective on how developers work was considered, then it was mostly measured
16 M. Felderer and G. H. Travassos

from a subjective perspective, e.g., by interviews or opinion surveys, or a “black

box” perspective by mining repository data or measuring the created development
artifacts. The chapter “Biometric Measurement in Software Engineering” introduces
biometric sensors and measure that provide new opportunities to more objectively
measure physiological changes in the human body that can be linked to various
psychological processes. These biometric measurements can be used to gain insights
on fundamental cognitive and emotional processes of software developers while
they are working, but also to provide better and more prompt tool support for
developers. Another human-related issue is the involvement of humans in empirical
studies, especially in experiments. On the one hand, it is normally difficult to recruit
a significant number of professionals for an empirical study, and on the other
hand, measurements are invasive. The chapter “Empirical Software Engineering
Experimentation with Human Computation” explores the potential of human com-
putation methods, such as crowdsourcing, for experimentation in empirical software
engineering.
Empirical methods rely on the collected data. However, the volume, velocity,
and variety of data in software products and processes have exploded during the last
years. Therefore, the new scientific paradigm of data science has gained much atten-
tion, also within software engineering. The chapter “Data Science and Empirical
Software Engineering” relates to traditional ESE and data science. It shows that both
paradigms have many characteristics in common and can benefit from each other.
Given large data sets, optimization is an important form of data analytics support of
human decision-making. Empirical studies serve both as a model and as data input
for optimization. The chapter “Optimization in Software Engineering: A Pragmatic
Approach” provides an overview of optimization in software engineering in general
and its value and applicability in ESE in particular. With increased automation,
uncertainty (due to the application of statistical models), and monitoring capabilities
in data-driven software engineering, also the role of simulation techniques becomes
more important. The chapter “The Role of Simulation-Based Studies in Software
Engineering Research” provides a guide to simulation-based studies in software
engineering. Bayesian data analysis is a means to embrace uncertainty by using
multilevel statistical models and making use of all available information at hand.
The chapter “Bayesian Data Analysis in Empirical Software Engineering: The
Case of Missing Data” provides an introduction to Bayesian data analysis and an
application example to empirical software engineering dealing with common issues
in ESE like missing data.
Extracting, aggregating, and synthesizing evidence from empirical studies is
essential for the development of scientific knowledge and the field of software engi-
neering. However, conducting secondary studies like systematic literature reviews
and aggregating evidence is still challenging. Conducting systematic literature
reviews (SLRs) is largely a manual and, therefore, time-consuming and error-
prone process. The chapter “Automating Systematic Literature Review” provides
strategies to automate the SLR process. Secondary studies often lack connection
to software engineering practice, which is essential to software engineering. The
chapter “Rapid Reviews in Software Engineering” presents the concept of rapid
The Evolution of Empirical Methods in Software Engineering 17

reviews, which are lightweight secondary studies focused on delivering evidence to

practitioners on time. Another approach to link to practice is to take grey literature
into account in empirical studies. The chapter “Benefitting from the Grey Literature
in Software Engineering Research” discusses the concept of grey literature in
software engineering and ways how to consider it in primary and secondary studies.
Considering that secondary studies are often used to support the evidence-based
paradigm, it is crucial to managing their threats properly. The chapter “Guidelines
for Managing Threats to Validity of Secondary Studies in Software Engineering”
provides guidelines for managing threats to validity of secondary studies in software
engineering. Evidence in software engineering is often rare and produced in both
quantitative and qualitative forms. It makes the synthesis of evidence, which is
an essential element in scientific knowledge creation, a challenging task. The
chapter “Research Synthesis in Software Engineering” provides an overview of
research synthesis methods in software engineering.
Society in general and funding agencies in particular put a stronger focus on
the impact of (software engineering) research. Therefore, open science and research
transfer are becoming essential topics in (empirical) software engineering. Open
science describes the movement of making any research artifact available to the
public and includes open access, open data, and open source. The topic is natural and
especially important in empirical software engineering to guarantee the replicability
of empirical studies. The chapter “Open Science in Software Engineering” reflects
upon the essentials in open science for software engineering to help to establish
a common ground and to make open science a norm in SE. Industry-academia
collaboration is one of the cornerstones of empirical software engineering. However,
close and sustainable collaboration with industry are key issues in the field.
The chapter “Third Generation Industrial Co-production in Software Engineering”
presents a seven-step industrial coproduction approach that enables deep and long-
term industry-academia collaboration.

5 Recommended Further Reading

Since 2000 research methodology has received considerable attention in the soft-
ware engineering research community. Therefore, plenty of literature is available
on empirical research methodology in software engineering. Molléri et al. (2019)
identified in a recent systematic mapping study 341 methodological papers on
empirical research in software engineering—and therefore, a complete overview
would exceed the scope of this book chapter. However, following the style of this
book chapter, we provide an overview of the available English text and special
issue books explicitly dedicated to empirical research methodology in software
engineering in chronological order of their publication.
Wohlin et al. (2000) published a book entitled “Experimentation in Software
Engineering,” which provides an overview of the core empirical strategies in
software engineering, i.e., surveys, experimentation, and case studies and as its
18 M. Felderer and G. H. Travassos

main content all steps in the experimentation process, i.e., scoping, planning,
operation, analysis and interpretation as well as presentation and package. The
book is complemented by exercises and examples, e.g., an experiment comparing
different programming languages. Consequently, the book targets students, teachers,
researchers, and practitioners in software engineering. In 2012 a revision of this
popular book had been published with Springer (Wohlin et al. 2012).
Juristo and Moreno (2001) published a book entitled “Basics of Software Engi-
neering Experimentation,” which presents the basics of designing and analyzing
experiments both to software engineering researchers and practitioners based on
SE examples like comparing the effectiveness of defect detection techniques. The
book presents the underlying statistical methods, including the computation of test
statistics in detail.
Endres and Rombach (2003) published “A Handbook of Software and Systems
Engineering. Empirical Observations, Laws, and Theories.” The book presents rules,
laws, and their underlying theories from all phases of the software development
lifecycle. The book provides the reader with clear statements of software and system
engineering laws and their applicability as well as related empirical evidence. The
consideration of empirical evidence distinguishes the book from other available
handbooks and textbooks on software engineering.
Juristo and Moreno (2003) edited “Lecture Notes on Empirical Software Engi-
neering,” which aims to spread the idea of the importance of empirical knowledge
in software development from a highly practical viewpoint. It defines the body of
empirically validated knowledge in software development to advise practitioners on
what methods or techniques have been empirically analyzed and what the results
were. Furthermore, it promotes “empirical tests,” which have traditionally been
carried out by universities or research centers, for application in industry to validate
software development technologies used in practice.
Shull et al. (2007) published the “Guide to Advanced Empirical Software Engi-
neering.” It is an edited book written by experts in empirical software engineering. It
covers advanced research methods and techniques, practical foundations, as well as
knowledge creation, approaches. The book at hand provides a continuation of that
seminal book covering recent developments in empirical software engineering.
Runeson et al. (2012) published a book entitled “Case Study Research in
Software Engineering: Guidelines and Examples,” which covers guidelines for
all steps of case study research, i.e., design, data collection, data analysis and
interpretation, as well as reporting and dissemination. The book is complemented
with examples from extreme programming, project management, quality monitoring
as well as requirements engineering and additionally also provides checklists.
Wieringa (2014b) published a book entitled “Design Science Methodology for
Information Systems and Software Engineering,” which provides guidelines for
practicing design science in software engineering research. A design process usually
iterates over two activities, i.e., first designing an artifact that improves something
for stakeholders, and subsequently empirically validating the performance of that
artifact in its context. This “validation in context” is a key feature of the book.
The Evolution of Empirical Methods in Software Engineering 19

Menzies et al. (2014) published a book entitled “Sharing Data and Models in
Software Engineering.” The central theme of the book is how to share what has been
learned by data science from software projects. The book is driven by the PROMISE
(Predictive Models and Data Analytics in Software Engineering) community. It is
the first book dedicated to data science in software and mining software repositories.
Closely related to this book, Bird et al. (2015) published a book entitled “The Art
and Science of Analyzing Software Data,” which is driven by the MSR (Mining
Software Repositories) community and focuses mainly on data analysis based on
statistics and machine learning. Another related book published by Menzies et al.
(2016) covers perspectives on data science for software engineering by various
authors.
Kitchenham et al. (2015) published a book entitled “Evidence-based Software
Engineering and Systematic Reviews,” which provides practical guidance on how
to conduct secondary studies in software engineering. The book also discusses the
nature of evidence and explains the types of primary studies that provide inputs to a
secondary study.
Malhotra (2016) published a book entitled “Empirical Research in Software
Engineering: Concepts, Analysis, and Applications,” which shows how to imple-
ment empirical research processes, procedures, and practices in software engineer-
ing. The book covers many accompanying exercises and examples. The author
especially also discusses the process of developing predictive models, such as defect
prediction and change prediction, on data collected from source code repositories,
and, more generally the application of machine learning techniques in empirical
software engineering.
ben Othmane et al. (2017) published a book entitled “Empirical Research
for Software Security: Foundations and Experience,” which discusses empirical
methods with a special focus on software security.
Staron (2019) published a book entitled “Action Research in Software Engineer-
ing: Theory and Applications,” which offers a comprehensive discussion on the use
of action research as an instrument to evolve software technologies and promote
synergy between researchers and practitioners.
In addition to these textbooks, there are also edited books available that are
related to special events in empirical software engineering and cover valuable
methodological contributions.
Rombach et al. (1993) edited proceedings from a Dagstuhl seminar in 1992
on empirical software engineering entitled “Experimental Software Engineering
Issues: Critical Assessment and Future Directions.” The goal was to discuss the state
of the art of empirical software engineering by assessing past accomplishments,
raising open questions, and proposing a future research agenda at that time.
However, many contributions of that book are still relevant today.
Conradi and Wang (2003) edited a book entitled “Empirical Methods and Studies
in Software Engineering: Experiences from ESERNET,” which covers experiences
from the Experimental Software Engineering Research NETwork (ESERNET), a
thematic network funded by the European Union between 2001 and 2003.
20 M. Felderer and G. H. Travassos

Boehm et al. (2005) edited a book entitled “Foundations of Empirical Software

Engineering: The Legacy of Victor R. Basili” on the occasion of V. R. Basili’s 65th
birthday, which covers reprints of 20 papers that defined much of his work.
Basili et al. (2007) edited proceedings from another Dagstuhl seminar in 2006
on empirical software engineering entitled “Empirical Software Engineering Issues.
Critical Assessment and Future Directions.”
Münch and Schmid (2013) edited a book entitled “Perspectives on the Future
of Software Engineering: Essays in Honor of Dieter Rombach” on the occasion
of Dieter Rombach’s 60th birthday, which covers contributions by renowned
researchers and colleagues of him.

6 Conclusion

In this chapter we presented the evolution of empirical software engineering in

five iterations, i.e., (1) mid-1960s to mid-1970s, (2) mid-1970s to mid-1980s, (3)
mid-1980s to end of the 1990s, (4) the 2000s, and (5) the 2010s. We presented
the five iterations of the development of empirical software engineering mainly
from a methodological perspective and additionally took key papers, venues, and
books into account. Available books explicitly dedicated to empirical research
methodology in software engineering were covered in chronological order in a
separate section on recommended further readings. Furthermore, we discuss—
based on the chapters in this book—trends on contemporary empirical methods in
software engineering related to the plurality of research methods, human factors,
data collection and processing, aggregation and synthesis of evidence, and impact
of software engineering research.

Acknowledgements We thank all the authors and reviewers of this book on contemporary
empirical methods in software engineering for their valuable contribution.

References

Akiyama F (1971) An example of software system debugging. In: IFIP congress (1), vol 71. North-
Holland, Amsterdam, pp 353–359
Araújo MAP, Monteiro VF, Travassos GH (2012) Towards a model to support studies of software
evolution. In: Proceedings of the ACM-IEEE international symposium on empirical software
engineering and measurement (ESEM ’12). ACM, New York, pp 281–290
Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized
algorithms in software engineering. Softw Test Verification Reliab 24(3):219–250
Baros MO, Werner CML, Travassos GH (2004) Supporting risks in software project management.
J Syst Softw 70(1):21–35
Basili VR (1993) The experimental paradigm in software engineering. In: Experimental software
engineering issues: critical assessment and future directions. Springer, Berlin, pp 1–12
Another Random Scribd Document
with Unrelated Content
 President of the Boston Charitable Irish Society, now in
its 172d Year.

Thomas P. Johnson, Noted Lawyer One

Hundred Years Ago.

Among the distinguished men who have

adorned the New Jersey bar few, in their day, were
held in higher repute for eloquence and extensive
legal knowledge, and especially for intellectual
vigor and versatility of talent, than Thomas P.
Johnson.
He was born about the year 1761. His parents
were Friends. His father, William Johnson, a
native of Ireland, emigrated to this country about
1747. He married Ruth Potts, of Trenton. Thomas
was their second child. When he was quite small
the family removed to Charleston, S. C., where the
father established a flourishing boarding-school
and gained much repute by his lectures on various
branches of Natural Philosophy. His fondness for
such studies seemed to have been inherited by the
son, who even in his later years continued to turn
his attention to them. The father died in the
South, after a residence of some years there. The
mother, with five children, returned to her native
state, and with the aid of her brother opened a
store in Trenton. There Thomas was placed an
[3]
apprentice to a carpenter and joiner. After
following this business some time he was
compelled, by a rupture of a blood vessel in the
lungs, to abandon it. He then engaged in teaching
in Hunterdom County and afterwards in Bucks
County, Pa.—later then in Philadelphia. For this
profession he had rare qualifications. Few men
had such powers of communication; few could so
simplify truth, and throw an interest around it to
captivate the youthful mind.
3. The annexed anecdote was communicated to the compiler
by a resident of Trenton: At one of the neighboring courts a
dispute arose between Johnson and his opponent respecting a
point of law, during which the latter remarked in a taunting,
derisive manner, “That he was not to be taught law by a
carpenter.” “May it please your honors,” replied Mr. Johnson,
“the gentleman has been pleased to allude to my having been a
carpenter. True, I was a carpenter. I am proud of it. So was our
Lord and Saviour. And I could yet, given a block of wood, a
mallet and a chisel, hew something that would very much
resemble that gentleman’s head. True, I could not put in brains,
but it would have more manners.”
While in Philadelphia a mercantile house took
him in partnership, and sent him to Richmond,
Va., where the firm opened a large store. There he
became well acquainted with Chief Justice
Marshall, and often had the privilege of listening
to the first lawyers in the Old Dominion. This
probably led to his turning his thoughts to the bar.
After a few years the loss of his store and goods by
fire caused him to return to the scenes of his
youthful days. He took up his residence at
Princeton, there married a daughter of Robert
Stockton, and entered his name as a student of
law in the office of the Hon. Richard Stockton. In
due time he was admitted to the bar as an attorney
and counselor. His career was brilliant. Whether
arguing points of law, or spreading a case before a
jury, he was always heard with fixed attention and
lively interest. So lucid was he in arranging and
expressing his thoughts, so quick to seize hold of
strong points in a case, and, when he pleased, so
well able to touch the chords of feeling, that he
rarely failed to produce an impression.
He was no indifferent student of the great
political questions. With the majority of the New
Jersey bar, he belonged to the Washington school,
and exerted all his energies in what he honestly
conceived to be his country’s real interests. For his
brethren of the New Jersey bar he cherished a
warm attachment, and they were forward in
evincing their high esteem of his worth. A few
years before his death a number of them obtained
the services of an artist and had a full-length
portrait of him executed. This now hangs over the
 judge’s chair in the court-room at Flemington. He
died March 12, 1838.

History of Newton, Gloucester County, N.

J.

Newton, Gloucester County, N. J., was early

settled by the society of Friends. The following
history was written by Thomas Sharp, the first
conveyancer and surveyor of the county:

“Let it be remembered. It having wrought upon ye minds of

some, Friends that dwelt in Ireland, a pressure having laid upon
them for some years which they could not get from under the
weight of until they gave upp to leave their friends and relations
there, together with a comfortable subsistence to transport
themselves and familys into this wilderness part of America, and
there by expose themselves to difficulties, which, if they could
have been easy where they were, in all probability might never
have been met with; and in order thereunto, sent from Dublin in
Ireland to one Thomas Lurtin a friend in London commander of
a Pink, who accordingly came, and made an agreement with him
to transport them and their familys into New Jersey, viz.; Mark
Newby and family. Thomas Thackarg and family, William Bate
and family, George Goldsmith an old man, and Thomas Sharp, a
young man, but no familys; and whilst the ship abode in Dublin
harbor providing for the voyage, said Thomas Lurtin was taken
so ill that could not perform ye same, so that his mate, John
Dagger, undertook it. And upon the 19 day of September, in the
year of our Lord, 1681, we sett sail, from the place aforesaid, and
through the good Providence of God towards us we arrived at
Elsinburg, in the country of Salem, upon the 19 day of November
following, where we were well entertained at the houses of the
Thomsons, who came from Ireland about four years before, who,
by their industry, were arrived to a very good degree of living,
and from thence we went to Salem, where were several houses
yet were vacant of persons who had left the town to settle in ye
country, which serve to accommodate them for ye winter,
proving moderate, we at Wickacog, among us, purchased a boat
of the Swansons, and so went to Burlington to the
commissioners, of whom we obtained a warrant of ye surveyor
general, which then was Daniel Leeds; and after some
considerable search to and fro in that then was called the third or
Irish tenth, we at last pitched upon the place now called Newton,
which was before the settlement of Phila; Pa; and then applied to
S,d Surveyor, who came and laid it out for us; and the next
Spring, being the beginning of the year 1682 we all removed from
Salem together with Robert Lane, that had been settled there,
who came along from Ireland with the Thomsons before hinted,
and having expectation of our coming only bought a lot in
Salemtown, upon the which he seated himself until our coming,
whose proprietary right and ours being of the same nature, could
not then take it up in Fenwicks tenth, and so began our
settlement; and although we were at times pretty hard bestead,
having all our provisions as far as Salem to fetch by water, yet,
through the mercy and kindness of God, we were preserved in
health and from any extreme difficulties.
“And immediately there was a meeting sett upp and kept at the
house of Mark Newby, and in a short time it grew and increased,
unto which William Cooper and family, that lived at the Poynte
resorted, and sometimes the meeting was kept at his house, who
had been settled sometime before. Zeal and fervency of spirit was
what, in some degree, at that time abounded among friends, in
commemoration of our prosperous success and eminent
preservation, both in our coming over the great deep as also that
whereas we were but few at that time, and the Indians many,
where by it put a dread upon our spirits, considering they were a
savage people; but ye Lord, that hath the hearts of all in his
hands, which cannot be otherwise accounted but to be the Lord
doings in our favor, which we had cause to praise his name for.
“And that the rising generation may consider that the
Settlement of the country was directed by an impulse upon
tranquility, but rather for the posterity yet should be after, and
that the wilderness being planted with a good seed, might grow
and increase to the satisfaction of the good husbandman. But
instead thereof, if for wheat it should bring forth tares, they
themselves will suffer loss.
“This narration I have thought good and requisite to leave
behind, as having had knowledge of things from the beginning.”

Anecdote of Washington.

Thatcher, in his Military Journal, gives a vivid

description of the sufferings of the troops during
“the hard winter of 1779–1780,” at Morristown, N.
J. He says:

“Morristown, January 1st, 1780. A new year commences, but

brings no relief to the sufferings and privations of our army. Our
canvass covering affords but a miserable security from storms of
rain and snow, and a great scarcity of provisions still prevails, its
effects being felt even at headquarters, as appears by the
following anecdote: ‘We have nothing but the rations to cook,
Sir,’ said Mrs. Thomson, a very worthy Irish woman, and
housekeeper to General Washington. ‘Well, Mrs. Thomson, you
must then cook the rations, for I have not a farthing to give you.’
‘If you please, Sir, let one of the gentlemen give me an order for
six bushels of salt.’ ‘Six bushels of salt for what?’ ‘To preserve the
fresh beef, Sir.’ One of the aids gave the order and the next day
his Excellency’s table was amply provided. Mrs. Thomson was
sent for, and told that she had done very wrong to expend her
own money, for it was not known when she could be repaid. ‘I
owe you,’ said his Excellency, ‘too much already to permit the
debt being increased, and our situation is not at this moment
such as to induce sanguine hope.’ ‘Dear Sir,’ said the good old
lady, ‘it is always darkest just before the daylight, and I hope your
Excellency will forgive me for bartering the salt for other
necessaries which are now on the table.’ Salt was eight dollars a
bushel, and it might always be exchanged with the country
people for articles of provision.”
 HON. JAMES CUNNINGHAM.

Of Portland, Me.

Vice-President of the Society for Maine.

 Cumberland County, N. J.

Baptist Church at Cohansey. As early as the year

1683 some Baptists from Tipperary, in Ireland,
settled in the neighborhood of Cohansey. The
most prominent persons were David Sheppard,
Thomas Abbott, and William Button.
Emigrants flocked into Cohansey from Ireland
and it is very probable that a Presbyterian Society
was formed about the year 1700 or earlier. Rev.
Robert Kelsey, who was from Ireland, used to
preach for the Baptists.
 THE FIRST CENSUS OF THE UNITED
STATES. SOME POINTED COMMENTS ON
THE MANNER OF TAKING SAME AND THE
RESULTS THEREOF. AN INTERESTING
PAPER ON A SUBJECT NOT HERETOFORE
TOUCHED BY THE SOCIETY.

BY MICHAEL J. O’BRIEN OF NEW YORK CITY,

AUTHOR OF “A GLANCE AT SOME PIONEER
IRISH IN THE SOUTH,” IN VOLUME VII OF
THE JOURNAL, AND OF MANY OTHER WORKS
OF HISTORICAL RESEARCH.

By an Act of Congress, entitled “An act

providing for the enumeration of the inhabitants
of the United States,” and which was signed by
President Washington on March 1, 1790, the
marshals of the judicial districts throughout the
United States were “authorized and required to
cause the number of the inhabitants within their
respective districts to be taken, omitting in such
enumeration Indians not taxed, and
distinguishing free persons, including those
bound to service for a term of years, from all
others.” These returns they were instructed to file
with the clerks of their respective District Courts,
who were directed to carefully preserve them.
This was the First Census taken of the
inhabitants of the United States, but it was far
from complete, for the reason that “heads of
families” only were recorded.
Eighteen months were allowed in which to
complete the enumeration. The census-taking was
supervised by the marshals of the several judicial
districts, who employed assistant marshals to act
as enumerators.
When the schedules were all gathered in, they
were turned over to the President, who, on
October 27, 1791, transmitted to Congress a
summary of the result, which was published in
what is now a very rare little volume that has not
been reprinted for public use. The original
schedules are contained in 26 bound volumes and
are still preserved in the Census Office. They form
a curious and most interesting collection, written
as they were by the assistant marshals, “on such
paper as they happened to have, and binding the
sheets together. In some cases printed blanks
furnished by the States were used, in others
merchants’ account paper, and now and then the
schedules were bound in wall paper.”
 A complete set of schedules for each State, with
a summary for the Counties, and in many cases for
towns, was filed in the State Department, but,
unfortunately, they are not now complete, “the
returns for the States of Delaware, Georgia,
Kentucky, New Jersey, Tennessee and Virginia
having been destroyed when the British burned
the Capitol at Washington during the War of
1812.”
In 1907, Congress authorized the Director of the
Census to publish, in a permanent form, the First
Census of the United States. “These schedules,”
says the Director of the Census, “form a unique
inheritance for the nation, since they represent for
each of the States a complete list of the heads of
families in the United States at the time of the
adoption of the Constitution. The framers were
the statesmen and leaders of thought, but those
whose names appear upon the schedules of the
First Census were in general the plain citizens,
who, by their conduct in war and peace, made the
Constitution possible, and by their intelligence
and self-restraint put it into successful operation.”
The First Census has a peculiar interest for
Americans of Irish blood or descent, for here we
find irrefutable evidence of the racial origin of a
large part of the people of the United States a few
years after the close of the Revolutionary war.
There is, of course, no standard, or fixed rule or
principle, by which an absolutely correct judgment
on the question of the racial composition of the
early inhabitants of the United States can now be
formed. The available statistics on the subject are
incomplete and confusing.
But, if names are to be accepted as a criterion,
those who examine the Census Returns, in
conjunction with the records of land grants, the
parochial registers, the Colonial Records that have
been collected and edited by the secretaries of
state, the court and church records, the
Revolutionary rosters, the old newspapers, the
Registers of Historical Associations, and other
similarly reliable records, must at once conclude
that a goodly percentage of the people were of old
Irish stock.
It must be borne in mind also that the Census
enumerators made no returns of unmarried
persons. This fact is important, when we place
beside it the statements of reliable historians that
the Irish exodus of the 18th century largely
comprised the youth of the country. From the
records which we have already quoted, we know
that thousands of the Irish youth became
indentured servants after their arrival in the
Colonies, and it is not likely that these people,
even though married while still in servitude, were
considered of sufficient importance by the census
enumerators to be included in the lists of the
“heads of families.”
One who examines these records for traces of
the Irish settlers will be surprised to find a most
inviting field of retrospect and research ever
widening before him. We do not need to wander
into the field of romance, as some writers
occasionally do, in search of proof that at the
beginnings of the Nation the Celtic element
figured to a larger extent than it has been credited
with at the hands of our historians.
There is so much of dry fact concerning them in
these hitherto obscure records as to make it a
perplexing thought for the investigator where to
begin upon an exposition of the part played by the
Irish Colonists and their descendants in shaping
the destinies of the future Republic. Theirs is
generally a prosaic story of trials bravely borne, of
victories snatched from rude nature in the face of
many difficulties. The pioneer settlers lived a
simple but rude life on the frontiers of civilization,
free from the artificialities of our latter-day mode
of living, but they made the wilderness which they
found to blossom as the rose and to become a fair
habitation for the generations that have succeeded
them.
Their story would hardly be worth relating were
it not for the fact that it affords the proof that men
and women of our race and blood were of the
“warp and woof” from which has been evolved the
new and composite race, miscalled “Anglo-Saxon,”
which has made this Western Hemisphere the
leader among the nations of the civilized world.
It is in the Census of the Southern States that
Irish names appear in the greatest numbers.
During the first half of the 18th century there were
large immigrations of Irish people to the
Carolinas, who spread themselves over an
immense area, reaching from the Santee river to
the eastern boundary line of Georgia, and as far
north as the dividing line of North Carolina.
Doubtless, the majority of those on the Census
Returns bearing Irish names were descendants of
those early settlers, rather than natives of Ireland.
The historian Lossing says: “Between the years
1730 and 1740 an Irish settlement was planted
near the Santee river in South Carolina, to which
was given the name of Williamsburg Township.
Up the Pedee, Santee, Edisto, Savannah and Black
rivers settlements spread rapidly, and soon the
axe and the plough were plying with mighty
energy, and from the North of Ireland such
numbers departed for Carolina that the
depopulation of whole districts was threatened.”
Williamsburg he calls a “hotbed of rebellion”
during the Revolutionary war. As soon as General
Francis Marion received his commission from
Governor Rutledge, we are told, “he sped to the
district of Williamsburg between the Santee and
Pedee to lead its rising patriots to the field of
active military duties.” (The rosters of General
Marion’s brigade contain a large number of Irish
names.)
Ramsay also refers to these Irish settlements
and deals with them at length in his History of
South Carolina. He says that the district was
named Williamsburg by an Irishman named
James, who came to the Colony with his father in
1733. It is now called Kingstree, and the county in
which it is situated is still named Williamsburg.
Sims’ Life of General Marion says: “the people
of Williamsburg were sprung generally from Irish
parentage. They inherited in common with all the
descendants of the Irish in America a hearty
detestation of the English name and authority.
This feeling rendered them excellent patriots and
daring soldiers wherever the British lion was the
object of hostility.”
 Other local historians of the South also refer to
the Irish settlements in this territory, which
continued with but intermittent intervals down to
the closing years of the 18th century. When we
turn to the Census schedules we find that the
statements of the historians are amply
corroborated.
In the Williamsburg district, that “hotbed of
rebellion,” we find mention of such Irish families
as Burke, Barrett, Biggen, Butler, Barron, Bryan,
Broaderick, Boland, Brady, Bradley, Cain,
Cummins, Connor, Cunningham, Collins, Conway,
Callihan, Cronan, Cantey, Corbett, Connell,
Castlelaw, Creed, Conally, Cochran, Dunn,
Dempsey, Dawson, Dollard, Downing, Donoho,
Donnally, Delaney, Dillin, Dailey, Dulon,
Dogharty, Earley, Flin, Foley, Fitzpatrick, Faning,
Gorman, Galaspy, Gibbons, Gallivant, Ganey,
Gowen, Gavin, Gallaher, Gill, Garven, Hagan,
Hart, Harrington, Hayes, Hainey, Joice, Jordan,
Kennedy, Keenan, Kelly, Kelty, Keen, Keefe,
Kerns, Kailey, Lynch, Leysath, Murphy, McCalvey,
McCartney, McGill, McFarlin, Manning,
McCormic, McKenny, McDowell, McKee,
McGinney, McCauley, McBride, McMullan,
Mulhollen, Mitchel, McConnell, McClare,
McIlveen, McGee, McFadden, Moore, McCottery,
McElroy, McMelly, McCleary, McDaniel,
McDonald, McCarthy, McCall, McSwain,
McWilliams, Morrison, McGraw, McCausland,
McCune, McElhaney, McFail, McClendon,
McGrath, McElduff, McAdams, McCoy, McCary,
McCain, Mahon, O’Brian, O’Neil, O’Cain,
O’Bannon, Phelon, Powers, Quinn, Rogers, Roach,
Riley, Reidy, Rial, Ryan, Sullivan, Shealds,
Swiney, Steele, Shannon, Timmons, Toole, Ward
and Walsh.
These are not all. Of the Murphys alone there
were in the Williamsburg district eleven families,
nine Kellys and several distinct families named
O’Brian, O’Neil, O’Bannon and McCarty.
We have selected only one of each name, in
order to show that the Irish settlements spoken of
by Lossing, Sims and Ramsay were drawn from
the South, East and West, as well as from the
North of Ireland. American historians are in the
habit of saying that the immigrants from Ireland
were mainly the so-called “Scotch-Irish” element
from the northern counties. While it is true there
were large settlements of Scotch Gaels in the
Carolinas, it is seen from the foregoing list that
nearly every county of Celtic Ireland was
represented in the “hotbed of rebellion” of South
Carolina.
 We see from the Census Returns that the
collectors wrote down the names phonetically in
most cases, having paid little or no attention to
spelling, or the use of capital letters where they
were needed. Some peculiar transformations in
names resulted from the carelessness or ignorance
of the enumerators. For instance, we find O’Neill
spelled “onailes” and “Ownaile”; O’Brien spelled
“Obrient” and “Obriant”; O’Farrell as “Opherl,”
and Casey as “Caycey”; Donovan is down as
“Dunnevant” and “Dunnaphant”; Doherty as
“Dehoitey” and “Dohoty”; Nolan as “noling”;
Sullivan as “Sellivent,” “Swillevaun” and
“Sewlovan”; Murphy as “Murff,” “Murph,”
“Murpry” and “Murfree”; Gallagher as
“Gollerhorn”; Flynn as “Phlyn” and “Fling”;
Kinsella as “Kincheloe”; McLaughlin as
“Maklafflin”; O’Hara as “Oharroe”; and O’Ryan as
“Orion.”
To the prefix “Mac” the enumerators, in many
cases, gave the sound of “Mag.” Thus we have such
name transformations as “Magnamee” and
“Magmanous.” We also find “Makmain” for
McMahon and “Muckleroy” for McElroy, and so
on. Fitzgeralds are down as “Fitzjarrel” and
“Jarrel”; Fitzpatrick as “Fitchparterack” and
“Pitch Patrick”; Reilly as “Royley” and “Royalley”;
Cassidy as “Casaty.” In some Southern city
directories we have come across the name of
“Pitch,” and we wonder if some of these are not
descendants of the “Pitch Patricks” or
Fitzpatricks!
It would be difficult to think of an old Irish
name that is not represented in the First Census,
and which was not, at some time or other,
translated into something very different in
appearance, and sometimes only partially
retaining the sound of the original name. In the
mutations of time, even these new names became
still further changed, so that many of the present-
day descendants of the Irish pioneers of the
Carolinas cannot be recognized as at all of the old
Gaelic race.
The most numerous Irish name on the First
Census of South Carolina is Murphy, there having
been 50 distinct families of that name, although
the 48 Kelly families gave them a close race. The
Gill and McGill families run nip and tuck with the
O’Neills and the Nealls. There were 34 of the
former to 33 of the latter. The O’Briens and
O’Bryans ran the gauntlet of many changes. The
Census enumerators failed to appreciate the
significance of the regal prefix “O,” so they wrote
down the name Obrient, Obriant, Bryan and
Briant. There were 53 of these in South Carolina
in 1790.
 HON. ALEXANDER C. EUSTACE.

Elmira, N. Y.

Ex-President New York Civil Service Commission.

 A worthy Member from New York State.

The Celtic “Macs” make a great showing. There are

upwards of one thousand of such families in all, the
“Macs” that are indigenous to Ireland being more
numerous than those that are supposed to be
exclusively native to Scotland. When we consider that,
in 1790, the total number of free white males of 16
years and upwards in South Carolina was only 35,756,
we can readily understand that one thousand heads of
families, with their wives and children, must have
constituted a large percentage of the total population.
An examination of the Census Returns indicates that
the average number of children to each of the Irish-
named families was five, so that, on a conservative
estimate, the “Macs” alone must have contributed
nearly 20 per cent to the population of South Carolina!
We find 40 Ward families, 26 McClure families, 26
McDaniels, 23 McKees, 22 McCoys, 20 McDowells, 19
Cauleys and McCauleys, 19 Mahons and McMahons,
18 McCalls, 17 McBrides, 17 McConnells, 16 McCarts
and McCartys, 12 McNeills, 11 McFaddens, and 10
McMullan families. There are also numbers of
McCormacks, McGees, McGowens, McGraws,
McGuires, McCrackens, McCanns, McCartneys,
McCarys, McClearys, McClendons, McCollums,
McElroys, McKennys, McKelveys, McLaughlins,
McManus, and many other similar Irish family names.
 There are 41 distinct families of Bradleys recorded,
29 Harts, 24 Sullivans, 28 Reynolds, 22 Canes and
Kains, 22 Hayeses, 22 Hendricks, 21 Dunns, 23
Connors and O’Connors, 21 Carrolls, 20 Logans, 20
Reillys and Royleys, 17 Dawsons, 14 Gilmores, 16
Manions and Mannings, 12 Hagans, 13 Walshes and
Welches, 13 Higginses, and 11 Lynch and Linch
families.
Among names that are common to Ireland and
England, there are 113 families named Moore in the
First Census of South Carolina, 80 Rogers and
Rodgers, 24 Morrows, 46 Collinses, 42 Butlers and 41
Fords. There are 43 heads of families named Mitchell,
41 Montgomery, 26 McDonald, 26 Cunningham, 18
Gillespie, 17 Cochran and 22 Kennedy families, some
of whom, no doubt, were Scotch.
Such names as Brady, Burke, Casey, Connelly,
Corbett, Cassidy, Callahan, Cleary, Cummings, Curry,
Daly, Doherty, Donnelly, Dempsey, Dowling, Duggan,
Doyle, Donovan, Ennis, Fitzgerald, Fogarty,
Fitzpatrick, Flinn, Garrett, Garvin, Gorman, Hogan,
Jordan, Kearns, Lyons, Malone, Mulligan, Madden,
Morrison, Nolan, O’Bannon, Quinn, Regan, Roach,
Ryan, Rutledge, Shannon and Shiels, as well as others
of ancient Irish origin, occur quite frequently in the
Census Returns, and in all parts of the state. Indeed,
we might say, with startling frequency, if they are
examined by that set of persons who are so fond of
telling us that the American people are of Anglo-Saxon
origin!
Besides the old Irish clan names, it is seen that a
very large number of the early inhabitants of South
Carolina bore names that have been common in
Ireland for centuries, although not all of Irish origin.
There are many Browns, Grays, Greens, Whites,
Griffins, Grimeses, Rices, Savages, Steeles, Glovers,
Raineys, Rays, Flemings, Staffords, Shaws, Gastons,
Parnells, Mileses, Reeds, Fergusons, Coxs, Courtneys,
Clarks, Carrs, Kerrs, Allens, Pattersons, Berrys, Hails,
Henrys, Morrises, Martins, Lowrys, Hollands,
Morrows, Jacksons, Laceys, Masseys and Leonards.
That some of these people were Irish seems beyond
doubt, particularly when we find such distinctive Irish
Christian names as Darby, Malachi, Patrick, Brian,
Cormac, Connor and the like. And when we see
O’Bryan Smiths, Patrick Smiths, and Michael and
Jeremiah Smiths, and other similar name
combinations, we can safely assume that in the
majority of cases they were of Irish origin or birth.
MEMORIAL TO JERSEY PRISON SHIP HEROES.

Since many prominent Irish Americans played an

important part in the success of the project for a fitting
memorial to the heroes of the British prison-ships of
the Revolution, it is fit that reference be made here to
this noble tribute and the manner of its
accomplishment.
The magnificent monument, costing in the
neighborhood of $200,000, was dedicated at Fort
Greene Park, Borough of Brooklyn, New York, on
November 14, 1908, in the presence of one of the most
distinguished and representative gatherings which
ever honored a like occasion in this country. Addresses
were made by President-elect Taft, by Governor
Hughes of New York, Secretary Luke E. Wright of the
Navy, and by other distinguished men, including
Patrick F. McGowan, Chairman of the Board of
Aldermen of the City of New York, and Daniel F.
Cohalan, Grand Sachem of the Tammany Society of
New York, an organization which did much toward the
success of the project and which contributed a
substantial amount.
The monument itself is one of the finest memorials
in the world. It stands in the center of a broad plaza,
reached by three flights of 34 steps each, 100 feet in
width. The height from the bottom of the plaza to the
top of the monument is nearly 200 feet. The top is
accessible and is reached by an electric elevator. The
monument is constructed of white granite from New
York state, and the steps from granite quarried at
Stonington, Penobscot Bay, Maine. The architects
were McKim, Mead and White, and the work is said to
have been the last of an extensive nature by the late
Stanford White.
The funds for the erection of this noteworthy tribute
were obtained through a government appropriation of
half the amount, a state appropriation of $25,000, an
appropriation by the city of twice that amount, and the
rest through subscriptions from societies and
historical and patriotic organizations. The Tammany
Society contributed the final $1,000 to complete the
required amount. This well-known organization was
the very first to secure a proper recognition of the
courage and patriotism of the prison ship martyrs
more than one hundred years ago.
Among the heroes of the British prison-ships were
many of Irish birth or extraction, and it is therefore a
subject for pride and satisfaction to us as a race that
their valor has at last been recognized, and especially
that an organization largely controlled by our people
has played so important a part in the accomplishment
of such recognition. Never in the history of the world
have prisoners of war been made the victims of such
unexampled cruelties as those practised on the
Americans by the British in the Revolution, and it is a
striking and never-to-be-forgotten commentary on
British methods toward their enemies in war that the
record of the prison-ships of the Revolution is deemed
by all the world a black mark on English history.
THE IRISH IN THE REVOLUTIONARY WAR.

Short Address Delivered by Master Karl Egan,

at the Iowa Opera House, Emmetsburg, Iowa,
March 17, 1909, During the Presentation of
the Irish Drama, “The Hero of Wicklow,”
under the Auspices of the Ancient Order of
Hibernians. Some Facts of Interest to the
People of Irish Birth or Ancestry.

Ladies and Gentlemen: On this important

anniversary, which is associated with so many
achievements of interest to the people of our race, it is
fitting to enquire what part the Irish took in the
Revolutionary War. What did they do for the cause of
human liberty at this most critical time in the world’s
history? All we ask is the truth. For some reason our
ordinary school histories have never given us any
credit for the prominent part our ancestors took in
that great struggle. What I shall say will bear the
closest historical investigation.
It was Patrick Henry, who, by his soul-stirring
speech, aroused the members of the Virginia Assembly
to a sense of patriotic duty. In 1776 he ran for governor
on the Independence ticket and carried that important
colony for the Revolutionary cause. Still, he claimed
that John Rutledge of South Carolina was the greatest
American orator of his time. The latter was also
elected president of South Carolina in 1776, on the
same ticket. John Rutledge and Patrick Henry were
both sons of Irishmen.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com