Experience of Industry Case Studies: A Comparison of

Multi-Case and Embedded Case Study Methods

Julian M. Bass Sarah Beecham John Noll
University of Salford Lero, the Irish Software Research University of East London
[email protected] Centre, [email protected]
University of Limerick, Ireland
[email protected]
ABSTRACT Embedded Case Study Methods. In Proceedings of 6th Interna-
This research comprises a methodological comparison of two tional Workshop on Conducting Empirical Studies in Industry
(CESI 2018) (ICSE’18). ACM, New York, NY, USA, 8 pages.
independent empirical case studies in industry: Case Study
https://doi.org/10.475/123_4
A and Case Study B. Case Study A, is a multiple-case study
involving a set of short-duration data collections with 46
practitioners at 9 international companies engaged in off-
shoring and outsourcing. Case Study B, in contrast, is a 1 INTRODUCTION
single case, participant observation embedded case study Case study research is one of several methodologies employed
lasting 13 months in a mid-sized Irish software company for empirical studies of software engineering in industry. Ac-
with geographically distributed software teams. Both cases cording to Yin, a case study is “an empirical inquiry that
were exploring similar problems of understanding the activi- investigates a contemporary phenomenon within its real-life
ties performed by various actors involved in scrum software context, especially when the boundaries between phenom-
development teams. In this study, we examine the findings enon and context are not clearly evident [36, p. 18].” Case
from both studies, the efficiency of the different case study studies can generate new hypotheses, and test existing hy-
methods and the contributions offered by each approach. We potheses [16]. Case studies “provide a systematic way of
adopted naturalistic research criteria to evaluate the two case looking at events, collecting data, analysing information, and
study approaches. We found that both multiple-case and reporting the results. As a result, case study researchers may
embedded case studies are suitable for exploratory research gain a greater understanding of why something happened as
(hypothesis development) but that embedded research may it did, and what might be important to investigate in future
also be more suitable for explanatory research (hypothesis research [35].”
testing). We also found that longitudinal case studies offer Under the broad heading of case study research are different
better confirmability; while multi-case studies offer better approaches to conducting case studies, including holistic that
transferability. We propose a set of illustrative research ques- focuses on an organization as a whole, and embedded, which
tions to assist with the selection of the appropriate case study examines different “units of analysis” within the broader
method. organizational context, possibly from different perspectives or
using different techniques [32, 36]. Case studies can also vary
CCS CONCEPTS by size along different dimensions, such as single vs. multiple
• Software and its engineering → Software creation and man- cases, or longitudinal studies that examine a phenomenon
agement; Software development process management; Col- over an extended period of time [30].
laboration in software development; Each methodology has strengths and weaknesses. This
research investigates similarities and differences between em-
KEYWORDS bedded case studies (longitudinal research conducted within a
Case Study Methods, Empirical Studies in Industry, Multi- single organisation) and multi-case studies (conducted across
case Study, Embedded Case Study, Cross-case Analysis multiple organisations). We want to identify criteria for se-
lecting deep embedded approaches or broad cross-company
ACM Reference Format: studies. While methods are sometimes dictated by circum-
Julian M. Bass, Sarah Beecham, and John Noll. 2018. Experi- stances (such as an opportunity to collaborate with a partner
ence of Industry Case Studies: A Comparison of Multi-Case and
company); we want to understand more clearly the strengths
and weaknesses of each approach.
Permission to make digital or hard copies of part or all of this work
for personal or classroom use is granted without fee provided that This leads us to identify the following research question:
copies are not made or distributed for profit or commercial advantage “What are the similarities and differences between embedded
and that copies bear this notice and the full citation on the first page.
Copyrights for third-party components of this work must be honored.
case study and multi-case study approaches in industry?” Fur-
For all other uses, contact the owner/author(s). ther we can identify two subsidiary research questions: “What
ICSE’18, May 2018, Sweden are the benefits and challenges of embedded and multi-case
© 2018 Copyright held by the owner/author(s). study approaches?” and “Under what circumstances should
ACM ISBN 123-4567-24-567/08/06. . . $15.00
https://doi.org/10.475/123_4 researchers select embedded and multi-case studies?”
 To answer these research questions, we undertake a cross- 2.1 Case Studies in Industry
case analysis of two previously published embedded and Empirical studies in field settings are conducted in order
multi-case studies performed by the authors. to explore software development processes and practices as
This paper is structured as follows: The next section perceived and understood by those involved in performing
presents previous research on empirical software engineering such work [23].
research methods, followed by a description of our research Hence, the focus of this research is on the case study
methods in Section 3. In Section 4, we describe our empirical method, which has been defined as:
findings, which is followed by a discussion in Section 5. Fi-
an empirical inquiry that investigates a contem-
nally, we provide conclusions and possible future directions
porary phenomenon within its real-life context,
in Section 6.
especially when the boundaries between phenom-
enon and context are not clearly evident [36].
In order to develop a triangulated and in-depth under-
2 EMPIRICAL STUDIES IN INDUSTRY standing in a contextual research setting, case studies often
A range of methods for conducting research have been iden- involve multiple data sources, including observation, inter-
tified, including: experiments, surveys, archival analysis and views, documents and archival records [14].
history [36]. Experiments rely on manipulating variables in Case studies tend to be either explanatory or exploratory
order to establish causal relationships. Surveys seek to ob- in nature. Explanatory case studies often address “how” and
serve the size and direction of relationships between variables “why” research questions [36]. Exploratory case studies, in
[9]. Archival analysis studies provide secondary documen- contrast, are often used to conduct an initial exploration
tary evidence, rather than primary sources [20]. Historical intended to generate new insights or propositions [14]. It has
research does not directly address contemporary issues. been suggested that case studies in software engineering are
These research methods can be employed to fulfil several exploratory because there is still a need to gather “basic
purposes: exploratory, descriptive, explanatory or improving knowledge about the human factors surrounding software
[29]. Exploratory research seeks new insights, generates new development and maintenance” [23, pg. 314].
hypotheses and ideas. Descriptive research articulates the Single case studies are typically used to inform on typical
current status of some phenomenon. Explanatory research experiences and provide a holistic, in-depth analysis of one
usually seeks to identify causal factors to explain a situation. setting, characterized by production of the rich and detailed
Improving research tries to refine or optimise some aspect of descriptions[36].
the studied phenomenon. A multiple case study design, in contrast, may potentially
The purpose of a survey is to produce quantitative or sacrifice some level of descriptive richness in each case, in
numerical descriptions of some aspects of the study popula- order to make comparisons across several settings [13]. It is
tion [21]. Information is usually collected by asking questions argued that multiple case designs are needed for creating a
with responses comprising the data to be analysed. Survey generalisable theory under the replication logic of positivist
information is usually collected from only a fraction of the case research [15].
population, that is a sample, rather than from every member
of the population. Surveys have been criticised for being 2.2 Case Study Trustworthiness
prone to sampling bias and limited geographical scope [34]. There is controversy about validity and reliability criteria for
Experimental approaches have long been advocated in com- research conducted in real-world settings [29]. On the one
puter science [37]. Experiments and quasi-experiments delib- hand research in industrial settings should aspire to the same
erately introduce some intervention in order to observe its standards of rigour as experimental and quasi-experimental
effect [9]. In its pure form, the application of interventions are methods. On the other hand, it is not typically possible to
randomised in experiments. However, in quasi-experiments, manipulate independent variables in order to observe the
treatments are not assigned randomly. impact on dependent variables in industrial contexts.
Qualitative research methods emphasize words, sounds and Conventionally there are four general criteria for determin-
images rather than numbers and quantitative measurement ing research quality [24, Chapter 11]:
in data collection and analysis [26]. In the context of software
∙ Truthfulness: do participants recognise the truth of the
engineering, there can appear to be a disconnect between
study findings,
research conducted and industry needs [5, 7]. Software en-
∙ Applicability: to what extent are the findings applicable
gineering research is complex due the intersection of issues
in other contexts,
around human behaviour during software development as
∙ Consistency: are the study findings repeatable with
well as human and computer capabilities [33]. Given the com-
similar participants in a similar context, and
plexity and variability of industry settings some researchers
∙ Neutrality: to what extent are the findings free from
call for more ‘context specific’ research that speaks directly
researcher bias.
to industry needs and does not set out to generalise results
[7]. These general criteria have evolved into four specific crite-
ria for experimental research: ‘internal validity’, ‘external
2
validity’, ‘reliability’ and ‘objectivity.’ Internal validity ad- Table 1: Comparison of multi-case and single-case, longitudi-
dresses the causal relationships between experimental vari- nal studies (adapted from Leonard-Barton [22]).
ables. External validity refers to the ability to generalise a
causal relationship between variables. Reliability applies to Activity Single-case, longi- Multi-case
the predictability, stability and accuracy of the study findings. tudinal
While, objectivity is usually set against subjectivity, where
Data Gathering
the concerns of individual participants dominate some sense
of collective judgement. Efficiency Low High
In contrast, and following the approach of [24, Chapter Objectivity Lower confirma- Higher confirma-
11], in this study, we adopt four criteria for exploring the bility (possible bility (but pos-
trustworthiness of naturalistic research findings: researcher bias) sible respondent
∙ Credibility, bias)
∙ Transferability, Pattern Recogni- Detailed Overall patterns
∙ Dependability, and tion
∙ Confirmability. Validity
These criteria attempt to address questions of validity and External Low transferabil- Higher transfer-
reliability in real-world research where experimental variables ity ability
are not easily manipulated to establish causal relationships Internal High credibility, Lower credibil-
[29]. esp. cause & ef- ity, esp. cause &
2.2.1 Credibility. The first research quality criterion to fect effect
be assessed relates to the “truthfulness” of the research. A Construct Can test sensitiv- Can test stability
positivist paradigm recognises a single external reality which ity to time across situations
researchers seek to uncover. However, naturalistic researchers
like [24] consider the realities created by groups, communities
and teams of people to be socially constructed. Hence, they
argue [24, Chapter 11] “truthfulness” is achieved when the lot of unusable data. Objectivity is threatened by researchers
research is carried out in such a way that the findings are becoming too deeply involved, and developing unconscious
found “credible” by researchers and study participants. biases. Pattern recognition tends to be microscopic in terms
of examination of highly context specific processes.
2.2.2 Transferability. The second criterion addresses the Whereas, there are some benefits to the multiple-site
applicability of research from one group of study participants method, where efficiency is relatively high, with a focused
to another. A positivist might assume that there is some data gathering effort, and pattern recognition is likely to be
universal truths, regardless of time and context, out there in good in terms of revealing patterns in the process (regardless
nature waiting to be discovered. But, many recognise that a of context). However there is still a threat to objectivity, with
group of study participants work within, and a guided by, a a danger of unconsciously accepting respondent bias, since
particular set of circumstances. Hence, in order to understand the researcher is more detached than in the single embedded
the likely application of research from one study group to longitudinal case study [22].
another, we need to understand the circumstances affecting When it comes to establishing validity, again we see several
that group. Transferability refers to the expectation that benefits to the multiple case study, that has relatively high
results from one context are applicable to another. generalizability as it views a variety of situations, where we
2.2.3 Dependability. The third criterion relates to the con- could not argue convincingly that observations made in any
sistency or repeatability of the research. Conventional studies single case study are generalizable [22].
are considered reliable if replication confirms the results. But There are gains to be made for internal validity in the
replication assumes that the precise circumstances faced by single case, with a good opportunity to establish cause and
a particular group, community or team of people can be effect in a longitudinal study. This is clearly lower in the
reproduced exactly. multiple case study, where confusion can exist in cause and
effect across the many settings, that are only visited at one
2.2.4 Confirmability. Confirmable research is neutral or point in time [22].
objective in terms of researcher interaction with the study When it comes to construct validity, both types of case
context. An independent observer should expect to reach have different strengths. In a single-case, longitudinal study,
similar conclusions in confirmable research. the sensitivity of construct measures (context and changes)
can be taken into account over the passage of time. Whereas
2.3 Summary the multiple case study offers a different form of stability,
Leonard-Barton compares the single case with the multiple where the construct can be assessed across situations [22].
case study, asserting that the efficiency of a single-case, lon- Table 1 summarizes Leonard-Barton’s comparison, using
gitudinal study is low, with a danger of data overload, and a the terminology of Section 2.2.
3
 Similar to Leonard-Barton [22], we focus on research method- Table 2: Case Study A– Demographic Information (n=46)
ology rather than the research topic under investigation
(Scrum activities performed). The motivation for conducting Company Company Sector Participant Roles

the study on scrum across various companies and contexts is A, India IT Service Provider Programme Manager, Senior Project Man-
ager, Team Member
secondary to the purpose of this research, which is to describe B, India Internet Engineering Manager, Product Manager
how the blending of two methodologies resulted in various C, India Software Service Development Manager
synergies, and unforeseen benefits and pitfalls.[22] Provider
D, India (offshore Software Service Project Manager, Product Owner, Scrum
provider to E) Provider Master (3), QA Lead, Team Member
3 METHODS E, UK, England Enterprise CRM Programme Manager, Project Manager, Di-
rector of Engineering
In order to discover the contrasting features of case studies
F, India Industrial Products Scrum Master
methods, a mixed method approach has been adopted com-
G, India IT Service Provider Engagement Manager
prising two case studies and a cross-case analysis. The first
H, India IT Service Provider Chief Technology Officer, Corporate Lead
phase (Case Study A) comprised a multiple case study of 46 Architect, General Manager Human Re-
practitioners from nine multinational companies; employing sources, Delivery/Programme Manager (3),
Project/Senior Project Manager (3), Scrum
a Glaserian grounded theory analysis of documentary sources, Master (2), Technical Analyst, Consultan-
t/Specialist (6), Team Member (9), Busi-
practice observations and interview transcripts. ness Analyst
In addition to Case Study A, during our second phase, I, UK, Scotland Customer Relationship Chief Operating Officer
we conducted a 14 month longitudinal embedded case study Management

(Case Study B) in a medium sized software development

company.
Finally, we employed a cross-case analysis to triangulate The main source of data in Case Study A were interviews
our findings. conducted, between January 2010 and March 2014, using
an open-ended semi-structured interview guide and were
3.1 Case Study A recorded and subsequently transcribed, as shown in Table 2.
The main source of data in this case study was a series of The semi-structured interview guide approach was em-
semi-structured face-to-face research interviews conducted ployed to allow some flexibility to adapt the interview ques-
with practitioners supplemented by some use of documentary tions to participants with different roles or responsibilities
evidence and workplace observation. [8]. The open-ended approach employed probing questions
to focus in more detail on issues raised by participants (and
3.1.1 Research Sites. Research sites were selected from a not included in the interview guide) during interviews.
set of multinational enterprises engaged in offshore or out-
sourced software development using a combination sampling 3.1.3 Data Analysis. The data analysis approach employed
approach comprising snowball and intensity sampling. in Case Study A involved open coding, constant comparison,
Snowball sampling used a network of former co-workers memo writing and theoretical saturation advocated in a
and other professional contacts to provide access to study Glaserian grounded theory approach [17].
participants, who then provided access to development teams Open coding, in this research, involves assigning a short
in different companies [28, p. 237], [27, p. 37]. This initial, word or phrase to symbolically represent the broader meaning
exploratory phase of the study, enabled data collection from of an action or item in the data [31]. The data analysis
a broad range of companies. software tool NVivo was used to record and formalise the
Intensity sampling was used to obtain a greater richness coding process [25].
and depth in the study, by accessing a larger number of Constant comparison is used to compare incidents that
interview participants with different responsibilities in the apply to each category, integrate categories and their proper-
same company or software development programme ([28] pp. ties, delimit the theory, and write the grounded theory [18,
234). Intensity sampling enabled triangulated perspectives pp.105]. Items in the data are compared within the same
from developers, quality assurance testers (QAs), project organisation or project team and with outside organisations
managers, development programme managers and corporate- and teams. During constant comparison analysis it is neces-
level executives sary to iterate back and forth between data collection and
analysis.
3.1.2 Data Collection. Case Study A made some use of Memo writing was used to clarify, refine and sharpen
secondary data in the form of corporate process guidelines, categories, which evolved as new transcript data is added [17,
project and development programme documentation, and Chapter 12]. Each memo comprised a short, often informal,
technical reports or white papers. Some of the participating essay on each topic including selected quotations to provide
companies also made commercially confidential details of illustrative primary evidence.
corporate agile practices, roles, policies and recommendations. Theoretical saturation occurs as the study evolves and
However, commercially confidential documentation was not the richness and detail of the analysis is enhanced by the
obtained from all of the companies, due to its commercial increasing number of study participants. Gradually, each
sensitivity. new participant provides evidence that is consistent with
4
the categories already identified and has less impact on the Next, Researcher 1 defined a set of codes and coded the
categorisation. text fragments accordingly. Then, Researcher 2 independently
coded the entire set of text fragments, using the same codes
3.2 Case Study B and associated definitions, The second researcher also noted
Case Study B was a moderate participant-observer study when no codes seemed to match a fragment, and when a
focusing on a development team from a medium-sized Irish- fragment appeared to be out of scope with the research
based software company that develops practice and lab man- question.
agement software for the optical industry. PracMed employs Finally, the two researchers discussed disagreements, re-
approximately seventy staff members in its software devel- sulting in some codes being removed or merged, and some
opment organization, including support and management new ones being introduced.
staff. PracMed’s annual sales approach e20 million, from
customers across the British Isles, continental Europe, Scan-
dinavia, North America, and China. 3.3 Cross Case Analysis
Case Study B focused on TeamA, whose responsibility is The case study approach is well established in software engi-
to tailor the company’s product for a large customer in North neering [30]. A cross case (or cross site) analysis is used to
America; the findings were validated by project management explore similarities and differences between cases [27]. We
team members from PracMed’s development organization can use multiple cases to establish the range of generality
(see Table 3). The members of TeamA are distributed over and conditions of applicability of each approach [18].
four countries on two continents, with up to eight hours Some authors have focused on synthesis of data from mul-
difference in timezones between locations. They are using tiple case studies [11, 12], three main approaches have been
Scrum to develop their software, with two weekly sprints. identified: narrative synthesis, thematic synthesis and the
cross case analysis [10]. Narrative synthesis focuses on using
Table 3: Case 2– Demographic Information (n=21) words and textual data to “tell a story” based on the case
study data. Thematic synthesis is a method for identifying,
PracMed Participant Role # participants analysing and reporting patterns within case study data.
Team A Product Owner 2 Cross case analysis facilitates the comparison of commonal-
Software Developer 5 ities and difference in the events, activities, and processes
Quality Assurance 1
Product Manager/Scrum 1
considered important for each case [19].
Master The approach we have employed for this research is a cross
case analysis from a methodological perspective [27]. We have
purposively selected the Case Study A and Case Study B in
3.2.1 Data Collection. The observations of TeamA in this order to contrast features of the methodology.
study were performed from January, 2016 to March, 2017.
Specifically, one of the authors observed approximately 200 of
TeamA’s Scrum ceremonies, including daily standups, sprint 4 RESULTS
planning, backlog grooming, and sprint retrospectives. Due Our main purpose here is not to present original findings
to team members being distributed across Europe and North from either case study itself, but rather to focus on the cross
America, the observations were made via video conference case analysis from a methodological perspective. However, es-
for each ceremony. The same author also conducted semi- tablish credibility for the methods used, previously published
structured interviews of each member of TeamA, which were results from each case study are summarised
recorded and transcribed. The interviews took approximately
one hour and followed an interview protocol available from
[6]. 4.1 Case Study A
The observer also made contemporaneous hand-written The multi-case study approach has been used to explore the
notes during both the ceremony observations and interviews. activities performed by product owners on large-scale off-
Finally, the interviewer summarized the interviews using a shore software development programmes [1, 2]. That research
mind-map, and presented the result to five interviewees in found product owners organised into hierarchical teams. For
an online workshop to validate the insights gained from the the large scale teams in that study, governance and risk as-
interviews. sessments were conducted in addition to more familiar agile
product ownership activities.
3.2.2 Data Analysis. Two researchers were involved in the
The multi-case study approach has also been used to ex-
data analysis, using a deductive approach, First, Researcher
plore the artefacts produced by cooperating software teams
1 extracted fragments of text from interview transcripts that
on large-scale development programmes [3]. In addition to
related to the research question. Then, Researcher 2 reviewed
producing working software, teams also produced detailed
these fragments, to identify any that seemed out of scope of
release plans in order to manage dependencies between coop-
the research question. This process was repeated to create a
erating agile teams.
set of relevant text fragments.
5
 Table 4: Research Questions Used in Case Studies Table 5: Key Features of Selected Case Studies

Research Question Study Criterion Case Study A Case Study B

“How do practitioners describe the tailoring of agile [2] Duration one day to 2 15 months
method roles and practices in large scale offshore weeks
enterprise software development programmes?”
Number of Subjects 46 9
“How do practitioners describe enhancement and [2]
Number of Companies 9 1
expansion of functions within the product owner role,
to meet the needs of large scale offshore enterprise Incident Observation Once Repeated
software development programmes?” Time Required to Con- Modest Extended
“How do the artefacts map to software development [3] duct Data Collection
processes used in large-scale offshore software devel-
opment programmes?” Table 6: Strengths and Weaknesses of Case Study Approaches
“How do these practitioner descriptions contribute [3]
to our understanding of artefacts in agile method Method Opportunities Challenges
tailoring in large-scale offshore software development
Embedded Repetition of Long time com-
programmes?”
Case Study events mitment
“What activities do Product Owners perform accord- [4]
Multi-case Triangulation Challenges to get
ing to the empirical literature?”
Study across companies access
“What activities do Product Owners perform in prac- [4]
tice?”
5 DISCUSSION
This research has investigated the question: “What are the
similarities and differences between embedded case study and
4.2 Case Study B multi-case study approaches in industry?” We contrasted
two case study approaches previously, and separately, con-
The embedded case study approach has been used to explore
ducted by the authors. In Case Study A a multi-case study
the activities of the Product Owner role in Scrum [4]. In
approach was used to investigate practitioner perceptions
particular, it was found that Product Owners perform a wide
of product ownership and development artefacts across nine
variety of activities, that can lead to stress and potential
international companies engaged in software development
conflicts of interest, resulting in negative impact on team
using agile methods.
performance.
Case Study B, in contrast, used an embedded case study
approach to explore practitioner perceptions of activities
4.3 Cross Case Analysis within the scrum master role.
The research questions from the case studies we have investi- We follow Lincoln and Guba [24] and use quality indicators
gated here are shown in Table 8. for empirical qualitative research: credibility, transferability,
The main features of the case studies we have investigated dependability and confirmability. We observe that Credibility
are summarised in Table 5. These features of each case study is enhanced when research questions are appropriate for the
were selected to investigate each research question. method selected. Dependability is enhanced when data is
We found that embedded case studies offer the opportu- collected with care and diligence as well as being subjected
nity to observe events (such as agile ceremonies: stand-up, to rigorous analysis.
retrospectives, etc.) repeatedly during a study period. This Drawing on the findings from our cross-case analysis, to
allows researchers to assess the frequency of best practices answer our second research question “What are the benefits
and observe teams under the varying project conditions they and challenges of embedded and multi-case study approaches?”
face over time. However, in order to gain the benefit of wit- we found that transferability is enhanced by using the multi-
nessing events repeatedly in the study company, researchers case approach whereas confirmability is enhanced by using
must expend a considerable time period. the embedded case study approach, as shown in Table 7.
In contrast, the multi-case approach provides the opportu-
nity to triangulate findings across companies. This creates the 5.1 Implications for Theory
opportunity to broaden the researcher’s repertoire of observed The case study method encapsulates a broad range of research
events and practices. However, it remains a challenge to get styles and approaches. Case studies can range from interpre-
access to companies, that tend to be secretive about their tive, constructionist and ethnographic qualitative research
practices in order to protect commercial advantage. These to positivist experimental quantitative studies. Case studies
opportunities and challenges are summarised in Table 6. in industry can be executed using a range of approaches.
6
 Table 7: Cross Case Analysis approaches taken in the case studies. The data collection in
Case Study A was performed in UK and India; while for Case
Method Strengths Weaknesses Study B, data collection was performed in Europe and North
America. The companies in Case Study A tended to be larger
Embedded Confirmability Transferability
multi-national enterprises, while Case Study B focused on a
Case Study
single medium-sized Irish company.
Multi-case Transferability Confirmability
Study

6 CONCLUSIONS
Table 8: Recommended Research Questions
In this research we have compared two approaches to con-
ducting case studies in industry: the participant observation-
Case Study Research Questions
Embedded Case Study How does the way [X] is performed evolve (over time)?
based embedded case study and a multi-case study involving
Why does [X] vary over time? practitioner interviews.
Multi-case Study How is [X] performed in different (units of analysis)? With access to sufficient resources, it would be attractive to
Why does [X] vary in different (units of analysis)?
undertake longitudinal studies across multiple cases. However,
such an approach requires multiple researchers (embedded in
multiple units of analysis) over a protracted period, requiring
Research questions and units of analysis must appropri-
an abundance of resources not available in our experience.
ately correspond to selected methods. Embedded cases lend
Case Study A used a combination sampling approach com-
themselves to detailed investigations of the project particular-
prising snowball and intensity sampling to investigate nine
ities, with potentially high levels of confirmability. Whereas
international companies engaged in geographically distributed
broad short-duration studies lend themselves to segmentation
software development predominantly using agile methods.
of research sites within a specific industry sector, business size
Case Study B, in contrast, was a participant observation
range or project style and hence can favour transferability.
embedded case study lasting 13 months in a mid-sized Irish
company producing medical practice management software
5.2 Implications for Research Practice
using a geographically distributed development team.
Gaining access to perform studies in industry is, in our experi- A methodological cross case analysis was employed to
ence, challenging. Researchers are not always in a position to compare commonalities and differences between Case Study
be selective about the type of access they have been granted. A and Case Study B. The cross case analysis contrasted
However, this research can inform the types of request for features of the two case study approaches.
access researchers make. If developing a partnership with a For the analysis in which we compared the two case study
specific company, then seeking longitudinal embedded access approaches, we selected four naturalistic qualitative research
is desirable. quality indicators: credibility, transferability, dependability
When an opportunity arises for access to industry, then re- and confirmability [24]. Using the findings from our cross-case
searchers need to derive Research Questions that correspond analysis, we found that whereas the multiple-case approach
to the type of access granted. Conversely, if negotiating in- enhances transferability, the longitudinal, embedded case
dustry access, ensure that the access requested corresponds study approach enhances confirmability. Conversely, ensuring
to the planned research questions. transferability is a challenge for the longitudinal, embedded
While it has been argued that all software engineering case study approach and confirmability is a challenge for the
case study research is exploratory [23, pg. 314], the enhanced multi-case approach.
confirmability of the embedded case study may makes this We found that the multiple-case and embedded case study
technique suitable for explanatory research. The longitudinal methods are suitable for exploratory research; both can be
nature of the embedded case study could offer an opportunity used to develop new hypotheses regarding the phenomena
for hypothesis testing. The multi-case study approach, if under investigation. However, we argue that the longitudinal
combined with shorter durations in each setting, is confined case study approach might also be suitable for explanatory
to exploratory research. research in which hypothesis testing is performed. However,
To answer out third research question: “Under what cir- research objectivity can be compromised by long-term inter-
cumstances should researchers select embedded and multi-case action with study participants. This can be alleviated to a
studies?” we can develop illustrative case study research certain extent through the involvement of researchers (not
questions, as shown in Table 8. directly involved in the data collection) testing the reliability
of the findings.
5.3 Limitations Further, we have identified research questions particularly
This research draws methodological conclusions from two suited to each approach. Research questions targeting change
previously published empirical case studies, conducted by and evolution of the phenomenon under investigation are
the authors, in industry settings. The cross case analysis suited to the longitudinal embedded case study approach.
is used to identify similarities and differences between the
7
Whereas research questions exploring variations of the phe- (1996). https://doi.org/10.3127/ajis.v3i2.383
nomenon under investigation within similar contexts are [14] S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. 2008.
Selecting Empirical Methods for Software Engineering Research.
better investigated by employing the multiple-case approach. In Guide to Advanced Empirical Software Engineering. Springer,
Future research will include extending the comparison London, 285–311. https://link.springer.com/chapter/10.1007/
978-1-84800-044-5_11 DOI: 10.1007/978-1-84800-044-5_11.
of the two case study approaches to include the positivist [15] Kathleen M. Eisenhardt. 1989. Building Theories from Case Study
criteria: internal validity, external validity, reliability and Research. The Academy of Management Review 14, 4 (1989),
objectivity. We also intend to consider mixed method ap- 532–550. https://doi.org/10.2307/258557
[16] Bent Flyvbjerg. 2006. Five misunderstandings about case-study
proaches, where the two case study types are supported with research. Qualitative inquiry 12, 2 (2006), 219–245.
cross case analysis to offer methodological triangulation. [17] B. G. Glaser. 1998. Doing Grounded Theory: Issues and Discus-
sions. Sociology Press, Mill Valley, USA.
[18] B. G. Glaser and A. L. Strauss. 1967. Discovery of Grounded
7 ACKNOWLEDGMENTS Theory: Strategies for Qualitative Research. Aldine, Chicago,
The authors are grateful to participants in Case Study A. IL., USA.
[19] S. Khan and R. VanWynsberghe. 2008. Cultivating the Under-
Thanks also go to IIIT-B and Company H who provided Mined: Cross-Case Analysis as Knowledge Mobilization. Forum
hospitality during several research visits. The research bene- Qualitative Sozialforschung / Forum: Qualitative Social Re-
search 9, 1 (Jan. 2008). https://doi.org/10.17169/fqs-9.1.334
fited in part from travel funding from the UK Deputy High [20] B. Kitchenham. 2004. Procedures for undertaking systematic
Commission, Bangalore; Science and Innovation Network. reviews. Technical Report TR/SE-0401. Department of Computer
We also thank the members of TeamA and for their gener- Science, Keele University and National ICT, Australia Ltd, Joint
Technical Report. http://www.scm.keele.ac.uk/ease/sreview.doc
ous and thoughtful collaboration on this study, and PracMed, [21] B. A. Kitchenham and S. L. Pfleeger. 2008. Personal Opinion
for allowing us to study their software development efforts. Surveys. In Guide to Advanced Empirical Software Engineering.
This work was supported, in part, by Science Foundation Springer, London, 63–92. https://link.springer.com/chapter/10.
1007/978-1-84800-044-5_3 DOI: 10.1007/978-1-84800-044-5_3.
Ireland grants 10/CE/I1855 and 13/RC/2094 to Lero - the [22] Dorothy Leonard-Barton. 1990. A Dual Methodology for Case
Irish Software Research Centre (www.lero.ie). Studies: Synergistic Use of a Longitudinal Single Site with Repli-
cated Multiple Sites. Organization Science 1, 3 (1990), 248–266.
[23] T. C. Lethbridge, S. E. Sim, and J. Singer. 2005. Studying
REFERENCES Software Engineers: Data Collection Techniques for Software Field
[1] J. M. Bass. 2013. Agile Method Tailoring in Distributed Enter- Studies. Empirical Software Engineering 10, 3 (July 2005), 311–
prises: Product Owner Teams. In Proc. IEEE 8th Int. Conf. 341. https://doi.org/10.1007/s10664-005-1290-x
on Global Software Engineering. IEEE, Bari, Italy, 154–163. [24] Y. S. Lincoln and E. G. Guba. 1985. Naturalistic Inquiry (1st
https://doi.org/10.1109/ICGSE.2013.27 ed.). SAGE Publications, Inc, Beverly Hills, Calif.
[2] J. M. Bass. 2015. How product owner teams scale agile methods [25] QSR International Pty Ltd. [n. d.]. NVivo 11 for Win-
to large distributed enterprises. Empirical Software Engineering dows. http://www.qsrinternational.com/nvivo/nvivo-products/
20, 6 (2015), 1525 – 1557. nvivo11-for-windows. ([n. d.]).
[3] J. M. Bass. 2016. Artefacts and agile method tailoring in large- [26] L. McLeod, S. G. MacDonell, and B. Doolin. 2011. Qualita-
scale offshore software development programmes. Information tive research on software development: a longitudinal case study
and Software Technology 75 (July 2016), 1–16. https://doi.org/ methodology. Empirical Software Engineering 16, 4 (Aug. 2011),
10.1016/j.infsof.2016.03.001 430–459. https://doi.org/10.1007/s10664-010-9153-5
[4] Julian M. Bass, Sarah Beecham, Mohammad Abdur Razzak, [27] M. B. Miles and A. M. Huberman. 1994. Qualitative Data Anal-
Clodagh Nic Canna, and John Noll. 2018. An Empirical Study ysis: An Expanded Sourcebook (2nd ed.). Sage Publications, Inc,
of the Product Owner Role in Scrum. In Proceedings of Interna- Thousand Oaks, CA, USA.
tional Conference on Software Engineering (ICSE 18). ACM. [28] M. Q. Patton. 2002. Qualitative Research & Evaluation Methods
[5] S. Beecham, P. O’ Leary, S. Baker, I. Richardson, and J. Noll. (3rd ed.). Sage Publications, Inc, Thousand Oaks, CA, USA.
2014. Making Software Engineering Research Relevant. Computer [29] C. Robson. 2011. Real World Research (3rd ed.). John Wiley
47, 4 (2014), 80–83. https://doi.org/10.1109/MC.2014.92 and Sons Ltd., Chichester, UK.
[6] Sarah Beecham, John Noll, and Mohammad Abdur Razzak. 2017. [30] Per Runeson, Martin H ost, Austen Rainer, and Bj orn Regnell.
Lean Global Project Interview Protocol. Available at http:// bit. 2012. Case study research in software engineering: Guidelines
ly/ 2nPxaXH . (2017). and examples. John Wiley & Sons, Hoboken, NJ, USA.
[7] L. Briand, D. Bianculli, S. Nejati, F. Pastore, and M. Sabetzadeh. [31] J. Saldana. 2015. The Coding Manual for Qualitative Re-
2017. The Case for Context-Driven Software Engineering Research: searchers Third Edition (3 edition ed.). Sage Publications Ltd,
Generalizability Is Overrated. IEEE Software 34, 5 (2017), 72–75. Los Angeles ; London.
https://doi.org/10.1109/MS.2017.3571562 [32] Roland W. Scholz and Olaf Tietje. 2011. Embedded Case Study
[8] S. Brinkmann and S. Kvale. 2014. InterViews: Learning the Methods. Sage Publications Inc., Thousand Oaks, CA, USA.
Craft of Qualitative Research Interviewing (3 edition ed.). Sage [33] C. B. Seaman. 1999. Qualitative Methods in Empirical Studies of
Publications, Inc, Thousand Oaks, CA, USA. Software Engineering. IEEE Transactions on Software Engineer-
[9] T. D. Cook, W. R. Shadish, and D. T. Campbell. 2001. Experi- ing 25, 4 (1999), 557–572. https://doi.org/10.1109/32.799955
mental and Quasi-Experimental Designs for Generalized Causal [34] M. Torchiano and F. Ricca. 2013. Six reasons for rejecting an
Inference (2nd ed.). Wadsworth Publishing, Boston. industrial survey paper. In 2013 1st International Workshop
[10] D. S. Cruzes, T. Dybå, P. Runeson, and M. Höst. 2015. Case on Conducting Empirical Studies in Industry (CESI). 21–26.
studies synthesis: a thematic, cross-case, and narrative synthesis https://doi.org/10.1109/CESI.2013.6618465
worked example. Empirical Software Engineering 20, 6 (Dec. [35] J. M. Verner, J. Sampson, V. Tosic, N. A. A. Bakar, and B. A.
2015), 1634–1665. https://doi.org/10.1007/s10664-014-9326-8 Kitchenham. 2009. Guidelines for industrially-based multiple
[11] D. S. Cruzes and T. Dybå. 2010. Synthesizing Evidence in Software case studies in software engineering. In 2009 Third International
Engineering Research. In Proceedings of the 2010 ACM-IEEE Conference on Research Challenges in Information Science. 313–
International Symposium on Empirical Software Engineering 324. https://doi.org/10.1109/RCIS.2009.5089295
and Measurement (ESEM ’10). ACM, New York, NY, USA, [36] R. K. Yin. 2009. Case Study Research: Design and Methods (4th
1:1–1:10. https://doi.org/10.1145/1852786.1852788 ed.). Sage Publications, Inc, Thousand Oaks, CA, USA.
[12] D. S. Cruzes and T. Dybå. 2011. Research synthesis in software [37] M. V. Zelkowitz and D. R. Wallace. 1998. Experimental models
engineering: A tertiary study. Information and Software Tech- for validating technology. Computer 31, 5 (May 1998), 23–31.
nology 53, 5 (May 2011), 440–455. https://doi.org/10.1016/j. https://doi.org/10.1109/2.675630
infsof.2011.01.004
[13] B. Doolin. 1996. Alternative Views of Case Research in Informa-
tion Systems. Australasian Journal of Information Systems 3, 2
8