Received: 29 November 2021

DOI: 10.1049/sfw2.12132

ORIGINAL RESEARCH
- -Revised: 26 March 2023 Accepted: 31 May 2023

- IET Software

Revisit security in the era of DevOps: An evidence‐based inquiry

into DevSecOps industry

Xin Zhou1 | Runfeng Mao1 | He Zhang1 | Qiming Dai2 | Huang Huang3 |

Haifeng Shen4 | Jingyue Li5 | Guoping Rong1

1
State Key Laboratory for Novel Software Abstract
Technology, Software Institute, Nanjing University,
Nanjing, China
By adopting agile and lean practices, DevOps aims to achieve rapid value delivery by
2
speeding up development and deployment cycles, which however lead to more security
Huatai Securities Co., Ltd., Nanjing, China
3
concerns that cannot be fully addressed by an isolated security role only in the final stage
State Grid Nanjing Power Supply Company,
of development. DevSecOps promotes security as a shared responsibility integrated into
Nanjing, China
4
the DevOps process that seamlessly intertwines development, operations, and security
Peter Faber Business School, Australian Catholic
University, Sydney, New South Wales, Australia
from the start throughout to the end of cycles. While some companies have already begun
5
to embrace this new strategy, both industry and academia are still seeking a common
Norwegian University of Science and Technology,
Trondheim, Norway understanding of the DevSecOps movement. The goal of this study is to report the state‐
of‐the‐practice of DevSecOps, including the impact of DevOps on security, practitioners'
Correspondence understanding of DevSecOps, and the practices associated with DevSecOps as well as the
He Zhang. challenges of implementing DevSecOps. The authors used a mixed‐methods approach
Email: [email protected] for this research. The authors carried out a grey literature review on DevSecOps, and
surveyed the practitioners of DevSecOps in industry of China. The status quo of Dev-
Funding information
SecOps in industry is summarized. Three major software security risks are identified with
Innovation Project and Overseas Open Project of
State Key Laboratory for Novel Software DevOps, where the establishment of DevOps pipeline provides opportunities for
Technology (Nanjing University), Grant/Award security‐related activities. The authors classify the interpretations of DevSecOps into
Numbers: KFKT2022A09, ZZKT2022A25; three core aspects of DevSecOps capabilities, cultural enablers, and technological en-
National Key Research and Development Program
of China, Grant/Award Number: 2019YFE0105500; ablers. To materialise the interpretations into daily software production activities, the
Research Council of Norway, Grant/Award recommended DevSecOps practices from three perspectives—people, process, and
Number: 309494; National Natural Science technology. Although a preliminary consensus is that DevSecOps is regarded as an
Foundation of China, Grant/Award Numbers:
62072227, 62202219; Key Research and extension of DevOps, there is a debate on whether DevSecOps is a superfluous term.
Development Program of Jiangsu Province, Grant/ While DevSecOps is attracting an increasing attention by industry, it is still in its infancy
Award Number: BE2021002-2 and more effort needs to be invested to promote it in both research and industry
communities.

KEYWORDS
software development management, software engineering

1 | INTRODUCTION Kanban [4] have been widely adopted in software develop-

ment. By seamlessly spreading the agile culture across devel-
Given the diverse customer demands and rapidly changing opment and operations, and by emphasising software quality
marketplace, a common desire about Software Engineering and collaboration between development and operation teams,
(SE) in industry is for agility in order to timely realise and/or DevOps [5] has emerged as a paradigmatic shift towards
adapt business value [1]. As a result, various agile methodol- evolving software at a continuous pace and streamlining all
ogies, such as Scrum [2], eXtreme Programming (XP) [3], and parts of the software lifecycle. It is crucial that software teams

-
This is an open access article under the terms of the Creative Commons Attribution‐NonCommercial‐NoDerivs License, which permits use and distribution in any medium, provided the
original work is properly cited, the use is non‐commercial and no modifications or adaptations are made.
© 2023 The Authors. IET Software published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology.

IET Soft. 2023;17:435–454. wileyonlinelibrary.com/journal/sfw2 435

436
- ZHOU ET AL.

have ownership and responsibility to deploy software changes

in DevOps [6], which allows the software to be delivered
quickly [7]. At such a frequent deployment and delivery (e.g. up
to 500 times a day in Facebook [8]), the software might not
undergo adequate security reviews [9]. In this context, Mac-
Donald from Gartner pointed out that [10]:1

Development, operations and security are

fundamentally intertwined and DevOps must
evolve to a new vision of DevOpsSec1. FIGURE 1 DevSecOps in Gartner Report [11].

As shown in Figure 1, Gartner describes the new and

updated services cycle through an iterative DevSecOps process describes the work we carried out after the preliminary find-
in their report, where security becomes a shared responsibility ings, in particular achieves five major contributions as below:
integrated into the entire DevOps process [11].
While software industry has been increasingly adopting 1. The state‐of‐the‐research on DevSecOps by academic
DevSecOps in their projects and the academia has began to pay literature is presented to justify the adoption of GLR.
attention to this new paradigm [12–14], a shared understanding 2. The major impacts of DevOps on software security are
of the DevSecOps movement underpinned by solid evidence is identified with the synthesised understanding of DevSe-
generally missing. As there is very little academic literature on cOps from the perspectives of capabilities and enablers.
DevSecOps, like the emerging stage of many other trendy topics 3. The recommended DevSecOps practices are classified in
in SE such as microservices [15], evidence has to first come in terms of people, process and technology.
the form of Grey Literature (GL), which is mostly produced by 4. The challenges of implementing DevSecOps and the
industry practitioners [16] and can serve as an important sup- contentious views on the relationship between DevOps and
plement to the shortage of academic literature [17]. DevSecOps are discussed to arouse the interests of both
Although DevSecOps has gained an increasing attention in research and industry communities.
academia [12–14], industry still leads this movement. Accord-
ing to our survey, the Chinese industry's attention to DevSe- The rest of this paper is organised as follows. Section 2
cOps is increasing year by year, but good practices are not introduces the background of DevSecOps and Grey Literature.
sufficient. We need to face the international industry to seek Section 3 describes the research questions and the methodol-
more practices and discoveries. Given the absence of academic ogy of our review. Sections 4 and 5 present the findings from
literature on DevSecOps, to gain the state‐of‐the‐practice of the synthesis of academic and grey literature respectively, fol-
DevSecOps, we carried out a Grey Literature Review (GLR) lowed by the discussion related to this review in Section 6.
on DevSecOps with reference to the guidelines [18]. From the Section 7 compares this study with the related secondary
215 grey articles retrieved by Google search engine, we ana- studies on DevSecOps and Section 8 shares lessons in con-
lysed the impacts of DevOps on software security and iden- ducting GLR and identifies potential threats to the validity of
tified three major challenges that DevOps brings to software our study. Section 9 draws the conclusions with a summary of
security including sacrifice of security for speed/agility, after- the findings, the implications, and the directions for future
thought in the process, and environment risks. Nevertheless, work.
the centralised and standardised DevOps pipeline also pro-
vides opportunities for performing security related activities.
We then classified the interpretations of DevSecOps into three 2 | THE VOICE ON THE SE
core aspects including DevSecOps capabilities, cultural en- COMMUNITY
ablers, and technological enablers. We further elaborate the
recommended DevSecOps practices from the three perspec- This section introduces grey literature and its use in SE as well
tives of people, process and technology in order to materialise as the background of DevSecOps from the perspectives of
the three interpretations of DevSecOps into daily software DevOps and software security respectively.
production activities.
This article is a significant extension of a prior conference
short paper version [19] that initially reports the limited find- 2.1 | Software security in DevOps
ings from the early stage of the review. This article extensively
Software security has always been a key concern of Chinese
enterprise segments, especially when cyber crime is accelerating
1
nowadays. If software security is not adequately addressed,
There exist three synonyms describing the integration of security into DevOps, that is, security breaches could result in massive losses in an enterprise.
SecDevOps, DevSecOps, and DevOpsSec. Although there might be some subtle
differences among them, for the ease of discussion, we use DevSecOps consistently For example, British Airways was attacked due to 22 lines of
throughout the paper. unsafe code, causing personal information leak involving
ZHOU ET AL.
- 437

approximately 380,000 customers in 2018 [20]. Kraemer's the research‐practice balance in SE research. In ref. [32], we
study [21] showed that programmers tend to neglect security identified five reasons why SE researchers considered GL in
issues when they are affected by certain external factors such as their studies, including seeking more related studies, avoiding
time pressure or high workload. When software is deployed publication bias, comparing different perspectives, under-
and delivered at a rapid pace by adopting DevOps, developers standing the views of the practitioner's community, and
are more susceptible to these external factors. If the changed exploring uncharted research areas. We also proposed a con-
software is deployed in a production environment without ceptual model to understand how GL works in SE research
undergoing sufficient security reviews, vulnerabilities are likely lifecycle.
to occur and consequently the risk of being attacked is high. With reference to Garousi's guidelines [31], Soldani et al.
Hence, many organisations and practitioners attempt to [15] conducted a systematic GLR on microservices with
integrate security into the DevOps pipeline by applying some rapidly evolving state‐of‐the‐practice in industry. They stated
protection practices, such as providing security training for that the efforts on microservices are still at an early stage in
developers and adopting some traditional security activities [9]. academia. They tried to bridge the gap between industry and
The concept of DevSecOps [22] opens up new horizons for academia by analysing 51 grey (industrial) studies published
holistically bringing security principles into the DevOps pro- from 2014 (when Lewis and Fowler enumerates the charac-
cess, which would help organisations in developing and teristics of microservice architecture) through 2017 (when
delivering high quality and more secure software. Various di- their review was carried out). Garousi et al. [18] further
mensions (culture [23], challenge [24, 25] etc.) of DevSecOps proposed guidelines for conducting Multivocal Literature
have been discussed in academia to form the full view of this Reviews (MLR) in SE by including GL and consulting the
concept. existing SLR guidelines, MLR guidelines and experience pa-
pers from other disciplines.

2.2 | Grey literature in Software Engineering

3 | RESEARCH METHODOLOGY
SE practitioners often share their knowledge and experience
through channels that are not as rigorous as academic (peer‐ This section elaborates the research design of this research
reviewed) publications, such as free online books and blogs with its processes depicted in Figure 2. The thin arrowed
[26–28]. Scientific information produced and published in this lines in this figure connect two different steps, while the
fashion is commonly referred to as grey literature. Because of artefacts produced by these steps are indicated by thick
the scarcity of academic literature on certain topics in SE, arrowed lines.
especially those that are trendy or industry driven, SE re- Our whole research process contains four main phases. In
searchers have recently started to pay more attention to grey Phase 1, through an ongoing survey of the status quo of the
literature. Shpilko et al. [29] proposed a model for GL by use of DevOps in the Chinese industry, we have learnt that
following Kepes's study [30] that divides literature into four Chinese practitioners have maintained a high degree of atten-
grey scales according to the difference in scope. tion to the specific practices of DevOps Security. Therefore,
As SE is a practitioner‐oriented engineering field, it is vital to the results of the survey have motivated us to understand the
establish connections between research and practice. Garousi dynamics of international practitioners. In Phase 2, a pre‐
et al. [31] stated that it is essential and useful to use GL to achieve search was first conducted in the middle of 2019 to get an

FIGURE 2 Research process of this review.

438
- ZHOU ET AL.

overview of this emerging new field and formulate our population and target invitees [33]. We hence chose the pur-
research questions (RQ2‐RQ4). We started Phase 3 of this posive sampling frame in this case.
GLR in October 2019 and reported the preliminary findings In recent years, we have insisted on publishing question-
[19] in SE community. The positive feedback from the com- naires online, spreading through the DevOps community and
munity motivated us to expand our review pools in SE. After InfoQ China3 and other technical communities in SE. In
systematic synthesis of the extracted data, we documented our addition, we are still publicising the questionnaires at confer-
findings in Phase 4. The following subsections elaborate these ences such as NJSD to attract more developers to participate in
phases of the review process. The research team consists of the survey. In order to collect as much data as possible, in the
three research students (one PhD and two master students) process of distributing the questionnaire, we also sent the
and their supervisors. All the research students have gained questionnaire to relevant practitioners who have participated in
extensive knowledge and experience in Empirical Software or downloaded the report we released, because they have a
Engineering through research training and participation in higher probability of participating investigation. Taking into
previous projects. account that the interviewed groups are all high‐income groups
with a strong interest in new technologies. In the way of
motivating participation in the questionnaire, we use the re-
3.1 | Research questions spondent's ability to obtain the survey report in the first time as
an incentive. This incentive method can also be better exclude
This study aims to address the following research questions: developers who have no interest and experience in DevOps,
and the collected data can be guaranteed to be of higher quality.
RQ1 What is the current situation of DevOps security in Respondents in this survey cover a wide range of in-
industry of China? dustries, positions, departments, scale of organisations,
RQ2.1 What are the impacts of DevOps on software DevOps experience, and cloud native usage. According to the
security in industry? questionnaire distribution form, it will be conducted through
RQ2.2 From what aspects do practitioners understand technical communities and related technical conferences.
DevSecOps? Distributed, the quality of participants is high (excluding
RQ2.3 Which practices in industry are associated with invalid questionnaires), and it is representative.
DevSecOps?

RQ1 is designed to seek more detailed information specific 3.2.2 | Questionnaire design
to the understanding and the use of DevSecOps in Chinese
enterprise segments. RQ2.1 is designed to examine how soft- We designed questionnaires to collect data (shown in Table 1).
ware security is affected by adopting DevOps in industry. In the questionnaire, we designed 12 questions from basic
RQ2.2 aims to explore the practitioners' understanding on the information, organisational information, performance, organ-
conception of DevSecOps. RQ2.3 steers our investigation to- isational culture, security practices, and security tools.
wards the practices that support DevSecOps.

3.2.3 | Questionnaire administration

3.2 | Opinion survey
For respondents who had participated in our survey and experts
In Phase 1, we seek more detailed information specific to the in the field of security, we distributed the invitations to partici-
understanding and the use of DevSecOps in industry of China pate in the survey via emails, which gave the reply period of
by means of questionnaires to both the DevOps practitioners 4 weeks for each sector of invitees. The invitees with no reply
and SE experts on the Global Conference of Software received were gently reminded of the survey 1 week after the first
Development (NJSD)2. NJSD conference participants round invitations. In distributing the invitations to their survey,
belonged to organisations from multinational companies and once any reply from them was received (either a decline to reply
some local Chinese companies. Since 2016, we have carried out or an automatic reply for unavailability), we immediately stop
the annual survey of ‘DevOps’ almost every year and reported sending the next round of email (reminder) to them.
it on the NJSD.

3.2.4 | Data synthesis and analysis

3.2.1 | Target invitees and sampling frame
In this survey, quantitative statistics is the most basic and
The survey instrument was designed from the respondent's commonly used method. In order to better measure the impact
perspective, which requires a clear identification of the of DevSecOps on an organisation's production activities, this

2 3
http://www.njsd-china.org. https://www.infoq.com.
ZHOU ET AL.
- 439

TABLE 1 DevSecOps questionnaire.

No. Question Choice options

1 How much time do you have in DevOps? A. Less than 6 months.
B. 6–12 months.
C. 1–2 years.
D. 3–5 years.
E. More than 5 years.

2 Which of the following titles matches you best? A. Development Engineer.

B. Testing Engineer.
C. Architect Engineer.
D. Automation or Tools Engineer.
E. Devops Cheng Shi.
F. SRE (Station Reliability Engineer).
G. Build or Release Engineer.
H. System Engineer.
I. Project Manager.
J. Technical Director.
K. Network Engineer.
L. Product Manager.
M. Operations Engineer.
N. Security Engineer.
O. Senior Management.
P. Others.

3 What kind of industry does your organisation belong to? A. Technology.

B. Internet.
C. Banking or Finance.
D. Education.
E. Communication.
F. Consultant.
G. Entertainment or Media.
H. Government.
I. Healthcare.
J. Retail.
K. Others.

4 Which of the following security practices does your organisation implement? A. Security and development teams collaborate on threat models.
B. Security tools are integrated into the development integration pipeline, so
engineers can be confident that they will not inadvertently introduce
known security issues into their code base.
C. Security requirements, both functional and non‐functional, are prioritised
as part of the product backlog.
D. Security experts evaluate automated testing and ask them to check for
changes in high‐risk areas of code (e.g. authentication systems, cryp-
tography etc.).
E. Before deploying, check the security policies related to the infrastructure.
F. Security personnel review and approve major code changes prior to
deployment.

5 Which of the following accord with your organisational structure? A. Centralised security capabilities, support delivery teams on demand.
B. Centralised security capabilities, with designated security experts for each
delivery team.
C. Decentralised security capabilities, each delivery team has a security
expert.
D. Other organisational structures.

6 Which of the following phases of the software development cycle does your A. Requirements phase.
organisation involve in security practices? B. Design phase.
C. Build phase.
D. Test phase.
E. Deployment phase.

7 Does your organisation have a professional security team? A. No, there is no professional security team.
B. No, there is no professional security team, but there are security man-
agement posts and personnel.
C. Yes, there is a special security management team and security supervisor.
(Continues)
440
- ZHOU ET AL.

T A B L E 1 (Continued)

No. Question Choice options

D. Yes, there is a high‐level security management organisation and a team of

security experts with perfect skills.
E. Yes, the team of security experts has made outstanding contributions to
the industry.
F. I don't know.

8 For DevSecOps, which of the following aspects do your organisation pay A. Whether the design comply with security standards and specifications.
more attention to? B. Code security.
C. Security of the third party open source library.
D. Automated configuration security.
E. Efficiency of automated security tools.
F. Security of cloud platform.

9 Which of the following phases does your organisation add automated A. Application architecture design phase
security testing? B. Code development phase
C. QA/test phase.
D. Deployment phase
E. Pre‐production phase
F. Production phase.
G. Continuous delivery of the whole process
F. Others (please add) ____________.

10 Which of the following is more in line with your organisation's security A. No security management.
management during software delivery? B. The security management of source code and dependent components is
carried out, and the security management is included in the test.
C. Perfect security scanning and testing toolchain, pipeline integration,
automatic security testing.
D. All participants are responsible for security.
E. With intelligent security delivery (code scanning, testing etc.) platform.
F. I don't know.
G. Others (please add) ____________.

11 Which of the following is more suitable for the security management of your A. No Security management.
organisation in the process of software operation? B, Security Monitoring covers some business scenarios, and periodic security
reporting is carried out.
C. Automatic security monitoring covers the whole business, the security
process is perfect, and the security problems are continuously sum-
marised and analysed.
D. Automatic security monitoring covers the whole business and infra-
structure, and some security events can be intelligent early warning and
self‐healing.
E. With an intelligent comprehensive security monitoring system.
F. I don't know.
G. Others (please add) ____________.

12 What security tools are being used in your organisation? Your answer: ________________________.

survey selected questions about the organisation's “security scoped our final research questions based on these knowledge.A
content, security practices, automation and security tools”. pre‐search was conducted via Google and Google Scholar with
the following string to retrieve online articles relevant to Dev-
SecOps as many as possible.
3.3 | Literature review
DevSecOps OR SecDevOps OR DevOpsSec OR
3.3.1 | Search process (DevOps AND Security) OR “Continuous
Security”
Developing a protocol is an important activity in a review pro-
cess as it gives details of the plan for the review [34], including Each research student quickly scanned the retrieved articles
process to be followed and allocation of reviewers to particular and regular meetings were held to discuss the results and
activities. In Phase 2, we first held a meeting to develop an initial manage discrepancies. In the end of this phase, the state of
protocol, and then conducted a pilot study to refine the protocol research and industrial adoptions of DevSecOps emerged.
until the final protocol was reached. We conducted pre‐search to According to our established understanding, we defined the
unified the knowledge of DevSecOps obtained from GL and research questions and developed the review protocols.
ZHOU ET AL.
- 441

After the protocol was finalised, we executed the search 3.3.3 | Data extraction
strategy defined in the protocol to identify relevant literature
both in industry and in academia with the identical search string. The data extracted from grey literature are basically similar, and
For searching grey literature in Phase 3, we set a continued here we only describe the process of extracting data from GL.
search period in order to collect as many relevant articles as Before the extraction, we redirected articles that clearly stated
possible using the identical search string. We noticed that some they were reproduced from other sources, which means that
of the highly relevant results retrieved in Phase 2 were no the extracted data is drawn from the original sources rather
longer available despite that their URLs still exist. To synthesise than from the reprinted ones. Table 3 lists the data extraction
as much evidence as possible for the research questions, we items that pertain to the research questions in addition to
build a final GL review pool by preserving all the relevant citation information. The three student researchers indepen-
articles retrieved in both pre‐search and continued search in dently read the full text of the articles assigned and extracted
PDF format. This pool consists of 78 articles from pre‐search the required data items. Any discrepancy in the extracted data
and 137 (63 + 74) articles from continued search after applying was discussed in routine meetings involving the student re-
inclusion and exclusion criteria listed in Table 2. searchers. The controversial questions were presented to their
supervisors who made expert decisions. All extracted data was
later cross‐checked by all the researchers involved after the
3.3.2 | Study selection extraction was done.

All the retrieved articles were divided into three groups, and
each of the three student researchers independently reviewed 3.3.4 | Data synthesis
two different groups, which ensures that each article was
reviewed by at least two different researchers. Any disagree- Both quantitative and qualitative methods were used to
ment was discussed in routine group meetings involving the synthesise the extracted data in order to answer the research
student researchers. The outstanding issues would be escalated questions. We applied thematic analysis in combination with
to their supervisors for final decision. narrative summaries. Coding was carried out for thematic
For the 174 grey articles collected in the pre‐search of
Phase 2 (in the middle of 2019), 78 were selected after applying
TABLE 2 Inclusion and exclusion criteria for grey literature.
the inclusion and exclusion criteria for grey literature defined in
Table 2. For the 115 more articles collected in Phase 3 (in Inclusion criteria
October 2019), 63 were selected after applying the same se- IN1 Written in English
lection criteria. Then for the 119 supplementary articles
IN2 Article content related to DevSecOps
collected in January 2020, 74 were selected. In total, 215 arti-
cles4 were retained for data extraction and synthesis. IN3 The full text of the article is accessible
Compared to academic literature, GL may display some Exclusion criteria
distinct characteristics. One is that a grey article may contain
EX1 Advertisement of vendor product or job recruitment
product marketing related material. Such an article typically
makes an objective statement of facts or describes a particular EX2 Books without full text access and videos
problem of interest, followed by recommended products from EX3 Product marketing articles
either the vendor itself or external organisations. To obtain as
EX4 New pages randomly redirected to the original literature
much evidence as possible in this review, we thoroughly read
the full text of such an article and selected it for inclusion if its EX5 Duplicated content
statement is neutral (not biased towards the recommended
products) and the product recommendation portion is less TABLE 3 Data extraction items.
than half of the article. Otherwise, the article would be
considered as a product teaser and excluded from the review. Data item RQ to be answered
For example, the majority of S [13] discusses the necessity of Title ‐
embedding security in DevOps and how to achieve security at Year ‐
speed, followed by promoting their own product at the end.
Accordingly, this article is selected as evidence into this review. The organisation where the website belong ‐
Another characteristic is that the link to a grey article may Terms related to ‘DevSecOps’ ‐
be randomly redirected to another URL whenever it is How DevOps affects software security? RQ2
accessed. To avoid infinite snowballing, we only included the
very first GL page in the review and ignore all subsequences. The definitions of DevSecOps RQ3

The principles of DevSecOps RQ3

The characteristics of DevSecOps RQ3

4
The list of all the selected articles and their URLs are available at https://figshare.com/
s/c90cd0c94c6b14ce6a15. Practices RQ4
442
- ZHOU ET AL.

analysis in our review. Whilst statistical analysis, shown in Chinese enterprise segments have a special security manage-
the figures and tables, was used for illustrating the distribu- ment team and security supervisor; 9.6% (23/239) of Chinese
tions of the selected articles, descriptive statistics was used to enterprise segments have a high‐level security management
illustrate practitioners' views from different perspectives of organisation and a well‐skilled team of security experts; only
DevSecOps. 2.9% (7/239) of Chinese enterprise segments have a team of
Table 4 shows an example of coding for thematic analysis, security experts, and have a good knowledge of the industry
where the three data items were initially coded with the labels outstanding contribution.
of ‘Challenges from the speed of DevOps’, ‘Challenges from the Automated security testing gradually covers the entire
agility in DevOps’ and ‘Ignore security for the sake of speed/ process, which can help companies find and solve secu-
agility’ respectively. After the thorough investigation and dis- rity problems as early as possible (as show in Figure 5).
cussion within the research team, these three codes were finally The survey results show that 27.2% (65/239) of companies can
consolidated into one label, that is, ‘Sacrifice security for speed/ introduce automated security testing in the application archi-
agility’. tecture design phase; 40.6% (97/239) of companies have added

4 | RESULTS FROM SURVEY (RQ1)

DevSecOps integrates security into each stage of DevOps. The

development, security, and operation departments work closely
together, emphasising that under the premise of controllable
security risks, it helps companies improve IT efficiency and
better realise DevSecOps.
This survey also set relevant questions for the security in
DevOps. From the reply, it is found that the security issues
concerned by different industries are different (as shown in
Figure 3). Among them, the entertainment industry has the FIGURE 3 Distribution of security issues concerned by different
largest difference, and its concern for the security of third‐ industries.
party open source libraries is only 6.70%. The Internet and
technology industries pay more attention to code security, but
the overall attention to all aspects of security is relatively
average.
We have received 239 effective feedback questionnaires
within 45 days after the questionnaire was released. Among
these replies, over 40% of Chinese enterprise segments
have introduced DevSecOps. The survey results show that
41.4% (99/239) of Chinese enterprise segments have intro-
duced DevSecOps; at the same time, 58.6% (140/239) of
Chinese enterprise segments have not introduced DevSecOps.
More than 40% of Chinese enterprise segments have
professional security teams, and security investment has
been valued by Chinese enterprise segments and devel-
oped rapidly. As shown in Figure 4, the survey results show
that 28.9% (69/239) of companies don't have a professional
security team; 28.5% (68/239) of companies do not have a
professional security team, but have security management
posts and personnel. At the same time, 29.7% (71/239) of FIGURE 4 Current situation of professional security team.

TABLE 4 An example of coding for thematic analysis.

Data item Initial code New code

‘DevOps pushes and modifies batches of code over very short time frames, which may far Challenges from the speed of Sacrifice security for speed/
outpace the speed at which security teams can keep up with code review’ DevOps agility

‘How do you fit in security while staying agile in DevOps’ Challenges from the agility in
DevOps

‘In their quest for speed, DevOps professionals potentially taking shortcuts that are Ignore security for the sake of speed
leaving their systems open to exploitation’
ZHOU ET AL.
- 443

automated security testing in the code development phase; all participants are responsible for security; 12.1% (29/239) of
47.7% (114/239) of companies have introduced automated companies have an intelligent security delivery (code scanning,
security testing in the QA/testing phase; other additions The testing etc.) platform.
stages of automated security testing are deployment (41.4%, Security operations are developing towards normal-
99/239), pre‐production (34.3%, 82/239), production (38.5%, isation, and the coverage is extended to business sce-
92/239) and the whole process of continuous delivery (28.5%, narios and infrastructure. The survey results show that
68/239). 18.8% (45/239) of companies have no security management
The security management of the software delivery in the software operation process; 34.3% (82/239) of com-
process covers source code and dependent components, panies' security monitoring covers some business scenarios
and the assembly line generally integrates automated and will periodically report on security; 21.0% (50/239) of
security testing. The survey results show that 19.2% (46/239) companies' automated security monitoring covers the entire
of companies have no security management in the software business, and the security processing process is complete,
delivery process; 29.4% (70/239) of companies conduct se- Security issues will continue to be summarised and analysed;
curity management on source code and dependent compo- 18.4% (44/239) of Chinese enterprise segments' automated
nents, and include security management in testing; 23.4% (56/ security monitoring covers the entire business and infra-
239) of companies have complete security scanning and testing structure, and some security incidents can be intelligently
tools Chain and assembly lines integrate automated security warned and self‐healed; 7.5% (18/239) of Chinese enterprise
testing; 15.9% (38/239) of companies incorporate security segments have an intelligent comprehensive security moni-
management into the entire process of R&D and delivery, and toring system.
The application of security tools is diversified, and the
penetration rate of containers and network security‐
related tools needs to be improved. In Figure 6, the sur-
vey shows that the host security tool NSFOCUS, the web
security tool AppScan and the code security tool Fortify are the
three most widely used security tools for Chinese enterprise
segments, accounting for 22.2% (53/239), 21.8% (52/239),
and 19.7% (47/239) respectively. Other security tools selected
by more than 15% are the code security tool Coverity (17.2%,
FIGURE 5 Automated security testing phase.
41/239) and the host security tool Nmap (16.3%, 39/239).

FIGURE 6 Current status of the use of security tools.

444
- ZHOU ET AL.

FIGURE 8 Partial distribution of organisations where the articles were

published.
FIGURE 7 Distribution of grey literature over years.

‘DevSecOps’. It is worth noting that among the 215 articles, 12

did not record the exact dates of publication, including 2 articles
that never mention ‘DevSecOps’. Figure 7 shows a clear
Finding 1: What Chinese enterprise segments are increasing trend of articles on DevSecOps with an exponential
considering is no longer ‘whether they need to growth in recent years, indicating practitioners' growing interest
embrace DevSecOps’, but ‘how to do a good job in adopting DevOps in practice recently. From 2016 to 2018, the
of DevSecOps implementation practice’. annual growth of GL on DevSecOps holds at approximately 20–
Finding 2: Along with the increasing improve- 30 more articles, while the number peaked at 92 in 2019 most
ment of DevSecOps strategic framework, the recently. In particular, the vast majority of the GL (72.5%) was
construction of related industries of China has published in the period between 2018 and 2019. It is worth
also been carried out rapidly, and the practice noting that the number of articles explicitly mentioning Dev-
effect of head industries such as finance, opera- SecOps follows a similar trend, which confirms that the concept
tors, communication and Internet is also gradu- of DevSecOps has been increasingly accepted by practitioners
ally improved. since its inception in 2012 [10]. For the articles that do not
Finding 3: Chinese enterprises segments have explicitly mention the term of ‘DevSecOps’, they are still rele-
made progress in project management, tool chain vant to this review and included in further analysis and discussion
usage and construction, security protection etc. as they pertain to a similar concept, such as ‘DevOpsSec’, ‘Sec-
‘Continuous automation testing’ is an important DevOps’, or the like.
focus of DevSecOps. We further analysed the distribution of organisations where
the articles were published in order to reveal their varying
degrees of interest in DevSecOps. As shown in Figure 8, the
selected articles are posted in 181 different organisations,
where TechBeacon5 contributes the most with four articles.
The results of the survey urge us to further understand the CISCO6, DXC technology7, Dark Reading8, SDTimes9 and
current status of DevSecOps in the industry through literature Checkmarx10 were each associated with three selected articles.
review. In addition, in the future, we can continue to iterate the The other 21 organisations are each associated with two
questionnaire regularly and conduct surveys on practitioners/ selected articles. Each of the remaining articles is only associ-
experts in the industry. ated with one of 154 organisations.

5 | RESULTS AND SYNTHESIS OF GREY 5.2 | Impacts of DevOps on software

LITERATURE security (RQ2.1)
This section first presents the demographic results of GL on We synthesised the impacts of DevOps on software security
DevSecOps and then discusses the answers to the research from the two perspectives of security risks and security op-
questions by synthesising the evidence. portunities. Security risks in DevOps can be further classified

5.1 | Study statistics 5

Homepage available at https://www.techbeacon.com/.
6
Homepage available at https://www.cisco.com/.
7

Figure 7 shows the distribution of the 215 grey articles published Homepage available at https://www.dxc.technology/.
8
Homepage available at https://www.darkreading.com/.
between 2013 and 2019, where the numbers in brackets and the 9
Homepage available at https://sdtimes.com/.
10
shaded portion indicate the articles mentioning the term of Homepage available at https://www.checkmarx.com/.
ZHOU ET AL.
- 445

into three categories through thematic synthesis, including profiles can also change quickly, making security a critical
sacrifice security for speed/agility, afterthought in the process, concern for DevOps initiatives. The development team rarely
and environment risks. Among the selected 215 articles, 142 have enough time to address all the security issues before the
(64.0%) are relevant to the impacts of DevOps on software product goes online, which means that there are potential se-
security and the vast majority are concerned with security risks curity risks on the Internet. All issues associated with a team's
(as shown in Figure 9). It should be noted that one article may structural division increase the development cycle time, delay
describe multiple impacts, hence the total number of the ar- delivery of valuable functionality or corrections, reduce
ticles in Figure 9 is more than 142. collaboration, and increase frustration and lack of trust within
the team. Therefore, a systematic approach is required to
improve the whole organisation's culture and structure,
5.2.1 | Security risks in DevOps fostering collaborative actions, in particular between security
and other divisions.
Sacrifice security for speed/agility
While many organisations have embraced the integrated Environment risks
approach to development and operations, they are often slow The affinity for DevOps teams to take to the cloud, however,
to include security within the DevOps framework. DevOps' creates new complications for security teams because con-
focus on speed/agility through cloud deployment, rapid ventional security measures mostly pertain to on‐premise
application development, frequently changing application fea- infrastructure. In addition, with the deployment of containers
tures and configurations, and speed‐prioritised and varying and microservices in cloud, DevOps teams also need to take
workloads, often leaves security teams flat‐footed and reactive. security considerations of these techniques into account.
More than one hundred of the selected articles point out that
DevOps practitioners degrade the priority of security since
they regard security as the biggest hurdle to rapid application 5.2.2 | Security opportunities in DevOps
development considering traditional security methods do not
fit the DevOps pipeline and are an inhibitor to DevOps agility. DevOps advocates that organisations build a centralised and
For instance, (S10) illustrates DevSecOps community survey’ standardised delivery pipeline, which can help the security team
result to present the phenomenon that ‘security is an inhibitor grasp what is being built so that they can take every oppor-
to DevOps agility’. It is highlighted in 20 articles that some tunity to inject various kinds of security activities into the
developers might simply use the same security credentials for pipeline. As discussed in (S3, S65), DevOps' high speed is
multiple assets simply for the sake of convenience, introducing achieved through a controlled and structured environment,
more risks to data protection. Particularly, (S138) points out instead of cutting corners and skipping important steps. Many
that developers are unlikely to keep tabs on different of the practices coming with DevOps, such as automation,
credentials. emphasis on testing, short feedback loops, improved visibility,
collaboration, consistent release practices and more, are a
Afterthought in the process fertile ground for integrating security and audit capability as a
The data sources show that security experts typically conduct built‐in component of a DevOps process.
tests at the end of the software development lifecycle, leading
to a situation where the security team works out of the
DevOps paradigm. Many companies find that increasing the 5.3 | Practitioners' understanding of
rate at which new iterations are released may cause teams to DevSecOps (RQ2.2)
bypass certain information security efforts. However, in a
world where code changes frequently, attack surfaces and risk Although there is a general consensus that DevSecOps is an
extension of DevOps [35], no commonly accepted definition
of DevSecOps has been formulated in both academia and
industry. Practitioners may possess different understanding of
DevSecOps that depends on their own professions, their
DevOps practice maturity levels, and the purposes of their
articles. Our discussion adopts the three core aspects of
DevOps, namely engineering capabilities, cultural enablers
and technological enablers, identified by Smeds et al. [36], to
look into the understanding of DevSecOps. Capabilities
represent the processes that an organisation is able to execute,
while the enablers allow a fluent, flexible, and efficient way of
working. Figure 10 categorise the 121 identified articles that
provide detailed information about their understanding of
FIGURE 9 Impacts of DevOps on software security. DevSecOps [36].
446
- ZHOU ET AL.

5.3.1 | DevSecOps capabilities the eventual cost of discovering errors by manual processes.
Moreover, automation helps organisation integrate security
The capabilities of DevSecOps include shift security to left and activities into SDLC (Software Development Life Cycle)
continuous security. Among the 68 articles relevant to the without slowing it down and enables developers to improve
capabilities, 47 of them focus on shift security to left and the code security without professional security knowledge. Secu-
remaining 21 focus on continuous security. Shift security to rity also needs to be integrated into the infrastructure since
left is not only about introducing security activities into the the greater scaled and more dynamic infrastructure enabled
early phase of development, but also about integrating security by containers have changed the software production
into the entire DevOps lifecycle. Furthermore, continuous environment.
security is more about continuous learning and continuous
improvement of projects and security delivery.
5.4 | Practices associated with DevSecOps
(RQ2.3)
5.3.2 | Cultural enablers
While practitioners' understanding reveals the fundamental
The cultural enablers list the traits that a DevSecOps team ideas and values characterising DevSecOps, the practices
should exhibit. In the 121 articles, 60 (49.6%) highlight the materialise them into daily software production activities [37].
importance of sharing responsibility, which means everyone in We summarised the extracted practices from three main per-
the value chain should be responsible for the security of the spectives: People, Process and Technology (PPT), referred to
end product. This shift of mindset makes the development and by some researchers as the “Golden Triangle” of organisation
operations teams to take some of the load off security and have operation fundamental principles [38]. Widely recognised as
a deeper understanding of how each discipline functions. The the three key elements for process improvement [39], PPT has
improvement of communication is also emphasised in 18 ar- been applied to the analysis of DevOps [40], cloud security
ticles as a smooth communication through the project cycle [41], information security [38], and other areas related to
facilitates cross‐departmental collaboration, which supports DevSecOps.
the creation of a security‐aware culture as a focus area of
DevSecOps and raise people's concerns on security
spontaneously. 5.4.1 | People

From the people's perspective, we identified the human factors

5.3.3 | Technological enablers in DevSecOps and summarised the main practices as shown in
Figure 11. Practitioners often discuss the human factors in
The selected GL shows that DevSecOps stresses the need for terms of culture, organisation, collaboration or communica-
automating security tasks since most of the practitioners re- tion. The main roles involved are DevOps teams and security
gard it as a technological enabler of DevSecOps. It is widely teams, where the relationship between developers and security
acknowledged that implementing automated security checks teams are most discussed. Specifically, security training and
in the DevOps pipeline will substantially reduce the time and security champion are the most discussed practices in the
people dimension of DevSecOps due to their positive effect on
breaking down traditional silos and improving communication.
Almost a third of the selected articles (72 out of 215)
mention security training, where the trainees include both

FIGURE 10 DevSecOps capabilities and enablers. FIGURE 11 People‐based DevSecOps practices.

ZHOU ET AL.
- 447

DevOps teams and security teams. For software and IT engi- development processes with the continuous delivery workflow
neers, security‐related training can equip them with the [42], which divides the processes in terms of artefacts and
guidelines for setting routines to improve security in the coding environment. Compared to the plan‐code‐build‐test‐operation
phase (S92), while the trainer can be a security specialist from model used in the conference version [19], the model in this
an external DevOps training organisation (S121) or from the article (as shown in Figure 12) is relevant to CI/CD and de-
internal security teams (S32). The training may cover attack's lineates the boundaries of processes. The workflow is divided
perspectives, practical hacking exercises, vulnerable applica- into five phases, where Pre‐Commit involves the activities
tions, and secure coding rules (S15, S146). For security teams, before the changed code is checked‐in to the source repository,
the objective of training is to be imbued with the DevSecOps followed by Commit phase that is triggered to build the
ethos and to learn about coding and APIs (S44). changed code, then transfer the binary files to the binary re-
Whilst security training is effective for DevOps teams and pository and subsequently to an Acceptance test environment.
security teams to share security responsibility and reduce Finally the change is deployed to Production after passing all
communication barriers, they are still two separated groups the previous phases. We separate the operations and produc-
with different specialities. To promote the communication and tion because the post‐deployment is a long‐term process which
collaboration between them, practitioners recruit or train se- is distinct from the production deployment.
curity champions in their teams. The security champions act as The number attached to every practice in Figure 12 de-
the voice of security for a given product or team, and they notes the number of the selected articles mentioning the
assist in the triage of security bugs for their team or area practice. In general, practitioners are more focused on Pre‐
(S121). From the 16 selected articles that introduce the concept Commit and Commit phases, which coincides with the
of ‘security champion’, 9 of them support nominating security concept of shifting security to left. As the most discussed
champions from DevOps teams to become the security con- practice in Pre‐Commit phase, threat modelling ensures that
science of the teams. Moreover, the assignment of security security is considered from the beginning of development. It
champions is also the first step towards creating a cross‐ identifies potential threats, estimates possible outcomes, and
functional team focused on application security and security creates a proactive mitigation strategy resulting in a solid threat
operations (S146). model (S24). However, threat modelling needs to be automated
Besides the DevSecOps teams, we observe that other or simplified because of its perceived slowness in DevOps
stakeholders especially executives also play an important role in (S15). To address security issues during coding, 14 articles
DevSecOps. Although only 9 selected articles mention the suggest including checks for defencive coding and security
importance of getting buy‐in from stakeholders, it is likely that vulnerabilities during peer code review, which could be a touch
these stakeholders are the key to a cultural change. The data on point for security teams to collaborate with developers (S215).
the latest security and data breaches and the consequences Another alternative to improve code security is security code
showing how the involved companies were negatively affected scanning with automated static analysis software testing
could make a strong case for convincing executives to get on (SAST) tools. We classified SAST into three practices: IDE
board with a cultural shift from DevOps to DevSecOps (S131). security checks, static code analysis, and deep SAST scan based
on the processes it is associated with. The IDE security checks
stress on the code consistency, maintainability, and clarity
5.4.2 | Process (S118), and the static code analysis looks for common coding
bugs and bug patterns to catch subtle logic mistakes and errors
From the process' perspective, we gathered the security prac- that could lead to runtime failures or security vulnerabilities,
tices and organised them according to the software while the deep SAST scan identifies security vulnerabilities

FIGURE 12 Process‐based DevSecOps practices.

448
- ZHOU ET AL.

through taint analysis, control flow, and other techniques. The security tasks is conducive to deliver security at developer's
dependency analysis and security unit tests are also conducted speed, and some practitioners even intent to automate every-
in Commit phase to reduce risks in the early stage. thing (S51, S103). In addition, security as code, infrastructure as
Following a successful build, the change is deployed to the code and compliance as code ensure that any code deployed is
test environment for acceptance tests. Although more than 10 secure and compliant, and that any deviation from this can be
articles introduce Dynamic Application Security Testing spotted early and fixed quickly (S121).
(DAST) and deep SAST scan as DevSecOps practices that can While speeding up the security delivery, some technical
be partly automated, they are still time‐consuming. Automated practices are being applied to address security challenges in the
attacks (S53) can simulate attacks on a running application by DevOps pipeline. Secrets in an Information Security envi-
executing a basic set of targeted automated pen tests against ronment include all the private information a team should
the system as part of the automated test cycle. The security know (e.g. a third party API). To establish a trusted connection,
checks and controls in production are mainly cloud security credentials, a certificate or an API token are necessary. Even
tests and configuration checks, which are automated to ensure with these precautions, however, handling secrets can be
the security of production environment. We find that the agility challenging, and can often become a source of error or even a
and speed of tests are less discussed after Commit phase, and security breach (S1). Secrets management can mitigate the risk
the automation is emphasised to reduce the manual work. of leaked credentials by making sure that accounts have only
In Operations phase, practitioners pay more attention to the privileges they need (S47). The management of configu-
vulnerability discovery and feedback. Red teams and Bug ration helps implement traceability of each code/configuration
bounties (S118) in this phase can demonstrate what is wrong change (S118), thus making it easier to identify the root cause
and provide a solution, creating a positive feedback loop be- of an issue and any deviation from immutable artefacts. Be-
tween security teams and the developers. Furthermore, orga- sides, vulnerability management continuously collects testing
nisations can address security issues using cyber threat results from the pipelines to assess and remediate vulnerabil-
intelligence solutions, which collect and process data auto- ities throughout the SDLC (S44) and it is responsible for
matically (S80, S106). protecting assets from known exploits and for identifying new
threats in software.

5.4.3 | Technology
6 | DISCUSSION
From the technology's perspective, we classified the practices
into two strategies in terms of their functions: speed up or This section first shows our observation of the current situa-
harden the DevSecOps processes, as illustrated in Figure 13. tion of DevSecOps in industry. Then discusses our decision
Technologies enable people to properly execute DevSecOps the process of GLR. Finally, we share the challenges to
processes, remove/relieve the need for the manual security implement DevSecOps.
activities to increase the delivery velocity and decrease the
attack surface to harden the pipeline.
Automation is the most popular strategy in DevSecOps 6.1 | Observation on DevSecOps in industry
mentioned in the vast majority of the articles (177 out of 215) and
also one of the pillars of DevOps [14]. As shown in Figure 13, 6.1.1 | Enterprises accelerate the pace of practice
Automation represents a set of automation‐related practices in
the DevSecOps processes. For instance, automation of recurring DevSecOps practice will no longer be limited to embedding
security tools into DevOps platforms, but will become an in-
dependent integration platform, and DevSecOps will take no‐
aware security as the core goal in the future, reducing the
interference of security tools to enterprise business production
and operation as much as possible.

6.1.2 | Practice needs to fit enterprise attributes

For most enterprises, DevSecOps often means drastic changes.

Not all A‐parties are suitable for direct application of Dev-
SecOps practice process, and the key security activities of
different industries are also differentiated, so it is necessary to
make further arguments according to their own organisational
development goals, cultural characteristics and business sce-
narios, and gradually figure out the enterprise's own security
FIGURE 13 Technology‐based DevSecOps practices. capability system.
ZHOU ET AL.
- 449

6.1.3 | Security is everyone's responsibility the answers are mainly code, or if the answers are generally short.
Therefore, we need to assess whether the content in GL is
Only when everyone participates in DevSecOps can security suitable for review when conducting the pilot research. As the
be ensured. In the process of implementing DevSecOps in results of the pilot study can meet the demand, we decided to use
mature enterprises, it is not difficult to find that the cultivation GL for the additional information out of academic literature.
of enterprise security culture is always listed as the top priority. The consistency of the evidence characteristics in GL and
Security and various teams need to participate in the contin- academic literature needs special attention for some RQs. In
uous construction and optimisation of the DevSecOps RD this study, to answer the questions about DevSecOps practice,
model, and continuously promote the theory and tool chain of the contents about practices were extracted and we found that
DevSecOps forward. GL has a higher level of abstraction whereas academic litera-
ture focuses more on specific technologies (e.g. the study on
self‐service cybersecurity monitoring [12]). Due to the incon-
6.2 | The choice of review method sistency of their evidence characteristics, and the difficulty of
the technology mentioned in the academic literature to be
After finding genuine reasons for undertaking a review, re- associated with DevSecOps practice, we tend to use GLR only
searchers need to decide which review method to use. Based to analyse and report.
on our study on GL and experience on conducting GLR, we
find three concerns of GL in this decision process (Figure 14):
possibility of introducing GL to a review and then using it as 6.3 | Challenges to implement DevSecOps
evidence, as well as the methods of using GL as evidence.
We first decide whether GL needs to be introduced to Similar to DevOps [47], DevSecOps also faces a number of
reviews. Three reasons, which motivated the inclusion of GL, challenges in order to be successfully implemented by an
were discovered in a tertiary study on MLRs in SE [43]. organisation. However, from the selected studies, only a
Similarly, our previous work [32], which was based on more handful of them (24 out of 215) describe the challenges to
reviews and the opinions from SE experts, summarised five implement DevSecOps, which can generally be categorised
reasons for including GL in SE reviews. Due to the papers into internal and external factors. In particular, culture, cost,
found in the academic literature retrieval stage can not provide and organisational structure are the three main internal factors.
effective evidence to answer our research questions, we Culture resistance is the most mentioned internal factor,
consider introducing grey literature to support the research. accounting for 11 out of 24 (45.8%) of the relevant articles.
With a loose organisational structure, GL may sometimes DevSecOps aims to remove the barriers between the three
not be used as evidence for reviews. For example, the contents in different teams and foster an atmosphere of collaboration in an
Stack Overflow, a community question and answer site used by organisation. However, it is hard to achieve these DevSecOps
many SE researchers (e.g. [44–46]), are not suitable for reviews if visions in reality due to the distinct core targets across these
teams. The development team prefers moving fast and
changing frequently, whereas the security team is in favour of
stability, which often leads to a conflict between them (S97). It
is also a new burden for developers to bear possible security
issues in mind when coding, which they rarely used to do
before. Their workload will also increase significantly by
learning extra security knowledge and skills.
High cost indicates the challenges due to economic reasons,
which are mentioned in 6 out of 24 (25.0%) of the relevant
articles. Many extra costs are the expense for the organisations
who decide to transform DevOps into DevSecOps. The costs
involved in changing the established DevOps pipelines are
usually unacceptable for small companies. Besides, Bogana
Dobran stated that ‘The performance and security re-
quirements of legacy resources create complications when fol-
ded into DevOps environments’ (S24). When integrating
security practices in legacy facilities without security consid-
erations, practitioners have to accept a relatively higher
complexity and investment.
Rigescent organisational structure is discussed in five ar-
ticles. Sarah Vonnegut pointed out that ‘if the burden of not
correctly securing DevOps environments isn't fully under-
stood by the board, it's impossible to expect the organisational
FIGURE 14 The decision process for review methods. structure to change’ (S131). All the stakeholders need to be
450
- ZHOU ET AL.

aware of the necessity of DevSecOps in order to support the expansion of DevOps aiming at integrating security processes
shift to DevSecOps. DevSecOps promotes ‘everyone is into DevOps life cycle through the collaboration of develop-
responsible for security’, but it is difficult to define everyone's ment, operations and security teams. Several DevSecOps char-
security boundary. Doug Drinkwater stated that ‘companies acteristics were generated from the articles and explained from
need to decide which roles have responsibility for security’ (S6). five aspects including culture, automation, measurement,
There are several connections among these internal factors. sharing, and shift security to the left. Five practices were
A company's culture and organisational structure are mutually discovered from the articles as well: (1) threat modelling and risk
reinforcing each other. For example, organisational structure assessments, (2) continuous testing, (3) monitoring and logging,
can be changed if security champions are appointed, and this (4) security as code, and (5) red‐team and security drills. Three
kind of role can accelerate the transformation of culture. benefits including shifting security to the left, automating secu-
Establishing what kind of culture and organisational structure rity, and security value as well as three challenges including
should take full account of cost factors as well. keeping up with DevOps, organisational challenges, and tools
In addition to the internal factors, we further identified three and practices were identified from the articles.
major external factors, which are experts, tools, and solutions. In [13], by reviewing two selected academic papers as well as
Lack of DevSecOps experts is discussed in nine articless. 11 grey articles from Google Search, Luís Prates et al. found nine
Cybersecurity Ventures reported that ‘there would be 3.5 relevant DevSecOps metrics reported by professionals: (1)
million cybersecurity job openings by 2021’ (S90). In this defect density, (2) defect burn rate, (3) critical risk profiling, (4)
context, Chris Carlson, VP of product management at Qualys, top vulnerability types, (5) number of adversaries per application,
pointed out that ‘the number of security practitioners knowl- (6) adversary return rate, (7) point of risk per device, (8) number
edgeable in DevSecOps is still low’ (S6). of continuous delivery cycles per month, and (9) number of is-
Lack of DevSecOps tools is explored in eight articles. sues during red teaming drills. Considering the tendency of
Although many DevOps tools and security tools have been DevSecOps and the exploratory nature of their study, they also
used in practice, DevSecOps requires new dedicated tools (S6). pointed out the needs for interviews and surveys with DevSe-
For example, the lack of visualisation tools makes it challenging cOps professionals.
to arrive at an informed decision on security in DevOps (S26). Table 5 compares this work with the two related work on
Lack of mature DevSecOps solutions is indicated in five DevSecOps in terms of topics, research questions, methods,
articles. Moving to DevSecOps may take a long time whilst search strings, databases, sample size, and results. Myrbakken's
there are still many phases that need to be inspected manually work [35] gave us a glimpse of DevSecOps from four aspects:
(S103) and for this reason some small companies are reluctant definition, characteristics, benefits, and challenges. Prates's work
to try DevSecOps (S6). [13] further explored the metric‐related issues in DevSecOps. In
Several links among external factors can also be identified. this study, we found in the pre‐search that until the middle of
DevSecOps experts contribute to the integration of different 2019 the number of academic papers related to DevSecOps
DevSecOps tools, and they can help improve the existing so- remained low and mostly had close collaboration with industry.
lutions to fit their company. The number of DevSecOps ex- As the absence of solid and comprehensive evidence from aca-
perts needed and DevSecOps tools integrated are influenced demic literature to answer our research questions (RQ2‐RQ4),
by the chosen solutions. we conducted a GLR with a broad search term focussing on how
Furthermore, there are intricate relationships among these practitioners understand DevSecOps and what types of practices
internal and external factors. The culture within an organisation are used in DevSecOps. With much more and up‐to‐date evi-
is profoundly affected by the employed experts and the adopted dence from the grey literature, our work not only presents a
solutions. Inherent company culture can also in turn significantly comprehensive vision of the state‐of‐the‐practice of DevSe-
impact the number of hired experts and the adopted solutions. cOps, but also proposes two frameworks, that is, the DevSecOps
Cost interacts with all external factors. Companies tend to capabilities and enablers model (Figure 10) and the DevSecOps
consider the cost of their practices at all times. Salaries for se- practices framework (Figure 11), which can be used as important
curity experts, the complexity of integrating security tools, and references for implementing DevSecOps.
the loss of using new solutions are all the issues that companies
need to consider if there is a limited budget before the official
institutionalisation of DevSecOps. Organisational structure can 8 | LIMITATION
be changed if required by the adopted solutions as well. Man-
agers can in turn choose the appropriate solution based on the We share our lessons in conducting GLR and summarise the
company's organisational structure. limitations of this study.

7 | RELATED WORK 8.1 | Lessons from exercising the grey

literature review process
Håvard Myrbakken et al. [35] analysed 52 artefacts returned by
Google search engine, which contained two academic research In this section, we review our experiences with the process of
papers and 50 pieces of GL (e.g. white papers, blogs, and articles). this GLR. When we conducted our review with reference to
In their research, DevSecOps was defined as a necessary the guidelines [18], we encountered some difficulties in the
ZHOU ET AL.
- 451

TABLE 5 Secondary studies on DevSecOps.

Article Håvard Myrbakken et al. [35] Luís Prates et al. [13] This study
Topic General DevSecOps DevSecOps Metrics General DevSecOps

RQs RQ1: How does the literature define RQ: Which are the most relevant RQ1: What are the impacts of DevOps on
DevSecOps? DevSecOps metrics? software security?
RQ2: What are the characteristics of RQ2: From what aspects do practitioners un-
DevSecOps? derstand DevSecOps?
RQ3: What are the main expected benefits and RQ3: Which practices are associated with
challenges of adopting DevSecOps? DevSecOps?
RQ4: Since it was first mentioned, how has
DevSecOps evolved?

Methods MLR MLR GLR

Search strings (“DevSecOps” OR “SecDevOps” OR DevSecOps, SecDevOps, Definition, DevSecOps OR SecDevOps OR DevOpsSec
(Search “DevOpsSec”) AND (“definition” OR Challenges, Metrics, Measuring, OR (DevOps AND Security) OR
terms) “characteristics” OR “challenges” OR Adoption “Continuous Security”
“benefits” OR “evolution”)

Databases Google Scholar, Google Search Google Scholar, Google Search, Google Scholar, Google Search, IEEEXplore,
IEEEXplore, ACM Digital Library, ACM Digital Library, SpringerLink,
SpringerLink ScienceDirect

Sample size 2 (academic) + 50 (grey ‐ first 5 pages on Google 2 (academic) + 11 (grey) 215 (three rounds of search)
Search)

Results The definition of DevSecOps; Nine relevant DevSecOps metrics. Study demographics of both academic and grey
Five DevSecOps principles; literature on DevSecOps (the tendency);
Five DevSecOps practices; Impacts of DevOps on software security (iden-
Three benefits gained from DevSecOps and its tification of security risks and opportunities
practices; in DevOps);
Three challenges an organisation faces from DevSecOps capabilities model (the practitioners'
DevSecOps; understanding of DevSecOps);
The evolutionary tendency of DevSecOps. DevSecOps practices framework (three types of
practices associated with DevSecOps).

phases of search and source selection. Our strategies may first we decided to exclude all of these articles because we
address these issues in the context of this review. thought that such information may skew the quality and
credibility of the articles. Nevertheless, we found some articles
with high‐quality evidence as well as advertisement. For
8.1.1 | Lessons learnt in the search phase example, we ever excluded the article (S15) because it recom-
mends a SAST tool developed by the author's company in the
We performed a pre‐search and a formal search with the same end of article, but anything else before that is unbiased.
search string with Google to collect GL. Although our intention Therefore, we double checked all of this kind of articles and
for pre‐search was to understand the problem and further refine finally decided to include an relevant article if its product teaser
our research questions, we found that some high‐quality sources is less than a half of the article.
were no longer available in the results of the later formal search,
which was attributed to Google search engine. Our solution was
to preserve the results of the pre‐search and combine the articles 8.2 | Threats to validity
from both the search processes. For future research with GL, we
suggest routine searches for a given period (days or maybe a With reference to the SLR guidelines [48, 49] and the guide-
week) and integrating all the results as the final literature set to lines for including GL in SE research [18, 32, 50], we identified
avoid the uncertainty of particular search engine at times. potential threats to the validity of our study and took actions to
mitigate them.
Construct Validity is concerned with the correct opera-
8.1.2 | Lessons learnt in the source selection tional measures for the studied concept [49]. Our goal is
phase defined by the research questions, which are answered based
on the categorisation schemes emerging from the evidence,
As GL is more diverse and less formatted than academic which was finalised through iterations with reference to some
literature, source selection can be particularly time‐consuming existing DevOps or security models [36, 51].
and difficult. Our major challenge in this phase is that some The lack of evidence in grey articles is another threat to
articles contain promotional material of software products. At construct validity. On the one hand, because of the
452
- ZHOU ET AL.

unstructured GL, they are largely based on authors' experi- not report the internal gaps in org strategy. Some companies
ences and opinions from practice, which could be regarded as that practice DevSecOps do not have the culture of blogging
evidence. On the other hand, some GL authors might cite and will not feature in the survey. It is true that such threats
other practitioners' statements. A serious limitation in grey exist, but they do not invalidate our results.
articles, which could be hardly fixed, is that their authors rarely Conclusion Validity is concerned with the repeatable
mention the sources of evidence or the systems they worked operations such as data collection procedure [49]. We devel-
on. Despite this, the evidence in forms of opinions and ex- oped the protocol that defines the data extraction strategy to
periences may also yield factual account and sound impression ensure the extracted data from the selected grey articles about
for the reason that the views of the practitioners, no matter DevSecOps. The review protocol was proposed by three stu-
whether repeated from others', can be essentially a status quo dent researchers, then reviewed and refined by their advisors
of the studied topic. A grey article could be also considered as and industry experts. The data extraction form was designed in
an unstructured interview or questionnaire for research pur- the protocol to secure the consistent extraction of evidence for
pose, which is common in ethnographic research to obtain each research question. Crosscheck was done after the inde-
more real and contextual information [52]. pendent data extraction by three student researchers. Any
Internal Validity is concerned with the causal relationship divergence was discussed in regular meetings with all the
whereby certain conditions are believed to lead to other con- authors.
ditions [49]. Considering that inclusion and exclusion criteria
may directly affect the quality of our data and further the an-
swers to our research questions, we planned fine‐grained 9 | CONCLUSIONS
procedures in order to secure the internal validity. We first
designed a rigorous search strategy and search process, then To achieve rapid value delivery, DevOps has been widely
applied the defined multi‐step selection process independently adopted in software industry. However, faster development
by three researchers. During this process, we also updated the and deployment cycles inevitably lead to more security con-
search strategy and selection process with necessary rework in cerns as the traditional security activities does not keep up with
certain scope. During the process, we always maintain uniform DevOps' pace and agility. DevSecOps was proposed to fill this
data extraction standards. Any disagreement was thoroughly gap by seamlessly intertwining development, operations, and
discussed until a consensus reached. security teams.
Although GL could help us discover the state‐of‐the‐ The study reported in this article contributes to the overall
practice of DevSecOps, it is difficult for researchers to eval- understanding of and insights into DevSecOps in practice by
uate GL [53, 54]. Our study analysed all the GL retrieved by reviewing much more comprehensive and up‐to‐date grey
Google in two search phases with the aim to offer a panoramic literature than ever.
view of DevSecOps in industry. While we realise the quality We articulate the impacts of DevOps on software security
differences among the included GL, there is no defined criteria from the perspectives of security risks and opportunities.
to properly assess and report them [29, 55], which results in the Based on the general consensus reached by practitioners that
conclusion of our pilot quality assessment with one fifth of DevSecOps is an extension of DevOps, we categorise the
included GL. Hence, we strongly call for the development of understanding of DevSecOps in the selected articles from
evaluation standards to be included in GL guidelines. three core aspects of DevOps, that is, DevSecOps capabilities,
In order to ensure the accuracy and truthfulness of the cultural enablers, and technical enablers. We also summarise
survey participants, we choose to publish the questionnaire in the existing typical DevSecOps practices in terms of people,
the soft industry technology community and related technical process, and technology. To arouse the discussion on DevSe-
conferences. We use the respondent's ability to obtain the cOps among practitioners and researchers, we also discuss the
survey report in the first time as an incentive, and this incentive ‘grey area’ between DevOps and DevSecOps as well as the
method can also be better exclude developers who have no challenges of implementing DevSecOps in industry in terms of
interest and experience in DevOps, and the collected data can internal and external factors.
be guaranteed to be of higher quality. We also ensured the This work aimed at evoking greater enthusiasm for Dev-
authenticity of the questionnaire answers through the following SecOps in academia and industry. As the concept emerging
measures: (1) Do a one‐tailed test for the answering time of the from software industry, the research on DevSecOps can offer
questionnaire, that is, exclude those whose answering time is the opportunity on reciprocal communication and collabora-
too short; (2) Set up a trap question to exclude those who fail tion between academics and industry. On the one hand, Tomas
the trap question. et al. [14] suggested a large‐scale empirical study by interviews
External Validity is concerned with the domain general- and surveys to be conducted to learn the state‐of‐the‐practice
ised by the study's findings [49]. Besides the GL, we also of DevSecOps in industry. On the other hand, ethnographic
surveyed the practitioners from Chinese or multinational methods, which can be used to explore the relationship be-
companies and experts from the SE community. This means tween human, process, technology and environment [56],
that our findings have universal significance in industry should be taken seriously in order to drive the investigation
(especially in industry of China). Moreover, GL articles may into DevSecOps in reality. We have always done this.
ZHOU ET AL.
- 453

AUT HO R CO N TR I BU T I ON S 7. Lwakatare, L.E., Kuvaja, P., Oivo, M.: Dimensions of devops. In: Pro-
Xin Zhou: Conceptualisation; Methodology; Writing – original ceedings of the 2015 International Conference on Agile Software
draft; Writing – review and editing. Runfeng Mao: Con- Development, pp. 212–217. Springer (2015)
8. Feitelson, D.G., Frachtenberg, E., Beck, K.L.: Development and
ceptualisation; Data curation; Writing – original draft; Writing deployment at Facebook. IEEE Internet Comput. 17(4), 8–17 (2013).
– review and editing. He Zhang: Conceptualisation; Funding https://doi.org/10.1109/mic.2013.25
acquisition; Methodology. Qiming Dai: Formal analysis; 9. Rahman, A.A.U., Williams, L.: Software security in devops: synthesizing
Investigation; Writing – original draft. Huang Huang: Formal practitioners’ perceptions and practices. In: Proceedings of the 2016
analysis; Investigation; Writing – original draft. Haifeng Shen: International Workshop on Continuous Software Evolution and De-
livery, pp. 70–76. IEEE (2016)
Resources; Validation; Writing – original draft. Jingyue Li: 10. MacDonald, N.: Devops Needs to Become Devopssec (2012). https://
Resources; Validation; Visualisation. Guoping Rong: Re- blogs.gartner.com/neil_macdonald/2012/01/17/devops-needs-to-become-
sources; Validation. devopssec/
11. MacDonald, N., Head, I.: DevSecOps: How to Seamlessly Integrate
Security into DevOps. Technical Report G00315283. Gartner (2016)
ACKN OW LE DG E ME N T S
12. Díaz, J., et al.: Self‐service cybersecurity monitoring as enabler for dev-
This work is supported by the Innovation Project of State Key secops. IEEE Access 7, 100283–100295 (2019). https://doi.org/10.
Laboratory for Novel Software Technology (Nanjing Univer- 1109/access.2019.2930000
sity) (KFKT2022A09, ZZKT2022A25), also jointly supported 13. Prates, L., et al.: Devsecops metrics. In: Information Systems: Research,
by the National Key Research and Development Program of Development, Applications, Education, pp. 77–90. Springer (2019)
China (No. 2019YFE0105500) and the Research Council of 14. Tomas, N., Li, J., Huang, H.: An empirical study on culture, automation,
measurement, and sharing of devsecops. In: Proceedings of the 2019
Norway (No. 309494), the National Natural Science Founda- International Conference on Cyber Security and Protection of Digital
tion of China (No. 62072227, No. 62202219), and the Key Services, pp. 404–411. IEEE (2019)
Research and Development Program of Jiangsu Province (No. 15. Soldani, J., Tamburri, D.A., Van Den Heuvel, W.J.: The pains and gains of
BE2021002–2). microservices: a systematic grey literature review. J. Syst. Software 146,
215–232 (2018). https://doi.org/10.1016/j.jss.2018.09.082
16. Banks, M.: Blog posts and tweets: the next frontier for grey literature.
CO NF L ICT O F I N T E RE ST STA T EM E NT In: Grey Literature in Library and Information Studies. De Gruyter
The authors declare that they have no known competing (2009)
financial interests or personal relationships that could have 17. Salleh, N., Mendes, E., Grundy, J.: Empirical studies of pair program-
appeared to influence the work reported in this paper. ming for cs/se teaching in higher education: a systematic literature re-
view. IEEE Trans. Software Eng. 37(4), 509–525 (2010). https://doi.org/
10.1109/tse.2010.59
DATA AVA I LA B I LI T Y STA T EM E NT 18. Garousi, V., Felderer, M., Mäntylä, M.V.: Guidelines for including grey
Data available on request from the authors. literature and conducting multivocal literature reviews in software engi-
neering. Inf. Software Technol. 106, 101–121 (2019). https://doi.org/10.
PER MIS SI ON T O R EP ROD U CE M AT E R I A LS 1016/j.infsof.2018.09.006
FRO M O TH ER S OU RC ES 19. Mao, R., et al.: Preliminary findings about devsecops from grey literature.
In: Proceedings of the 20th International Conference on Software
Upload a copy of our conference paper with the designation Quality, Reliability and Security, pp. 450–457. IEEE (2020)
‘Additional File for Review but NOT for publication’, and 20. Klijnsma, Y.: Inside the Magecart Breach of British Airways: How 22
specify which conference our original article was presented at Lines of Code Claimed 380,000 Victims (2018). https://www.riskiq.
below: 20th International Conference on Software Quality, com/blog/labs/magecart-british-airways-breach/
21. Kraemer, S., Carayon, P., Clem, J.: Human and organizational factors in
Reliability and Security (QRS2020).
computer and information security: pathways to vulnerabilities. Comput.
Secur. 28(7), 509–520 (2009). https://doi.org/10.1016/j.cose.2009.04.006
O RC ID 22. Mohan, V., Othmane, L.B.: Secdevops: is it a marketing buzzword?‐
Xin Zhou https://orcid.org/0000-0002-3263-1275 mapping research on security in devops. In: Proceedings of the 11th
Huang Huang https://orcid.org/0000-0002-1296-4363 International Conference on Availability, Reliability and Security, pp.
542–547. IEEE (2016)
23. Sánchez‐Gordón, M., Colomo‐Palacios, R.: Security as culture: a sys-
R EFE REN CE S tematic literature review of devsecops. In: Proceedings of the IEEE/
1. Cohen, D., Lindvall, M., Costa, P.: An introduction to agile methods. Adv. ACM 42nd International Conference on Software Engineering Work-
Comput. 62, 1–66 (2004) shops, pp. 266–269 (2020)
2. Schwaber, K., Beedle, M.: Agile Software Development with Scrum, vol. 24. Akbar, M.A., et al.: Toward successful devsecops in software develop-
1. Prentice Hall, Upper Saddle River (2002) ment organizations: a decision‐making framework. Inf. Software Tech-
3. Beck, K.: Extreme Programming Explained: Embrace Change. Addison‐ nol. 147, 106894 (2022). https://doi.org/10.1016/j.infsof.2022.106894
Wesley (2000) 25. Rajapakse, R.N., et al.: Challenges and solutions when adopting devse-
4. Ahmad, M.O., Markkula, J., Oivo, M.: Kanban in software development: cops: a systematic review. Inf. Software Technol. 141, 106700 (2022).
a systematic literature review. In: Proceedings of the 39th Euromicro https://doi.org/10.1016/j.infsof.2021.106700
Conference on Software Engineering and Advanced Applications, pp. 26. Schöpfel, J.: Observations on the future of grey literature. Grey J. 2,
9–16. IEEE (2013) 67–76 (2006)
5. Ebert, C., et al.: Devops. IEEE Software 33(3), 94–100 (2016). https:// 27. Glass, R.L.: Software Creativity 2.0. Developer.* Books (2006)
doi.org/10.1109/ms.2016.68 28. John, N.A.: The social logics of sharing. Commun. Rev. 16(3), 113–131
6. Lwakatare, L.E., et al.: Devops in practice: a multiple case study of five (2013). https://doi.org/10.1080/10714421.2013.807119
companies. Inf. Software Technol. 114, 217–230 (2019). https://doi.org/ 29. Adams, R.J., Smart, P., Huff, A.S.: Shades of grey: guidelines for working
10.1016/j.infsof.2019.06.010 with the grey literature in systematic reviews for management and
454
- ZHOU ET AL.

organizational studies. Int. J. Manag. Rev. 19(4), 432–454 (2017). https:// International Symposium on Empirical Software Engineering and Mea-
doi.org/10.1111/ijmr.12102 surement, pp. 1–6. IEEE (2019)
30. Kepes, S., et al.: Publication bias in the organizational sciences. Organ. 44. Zou, J., et al.: Which non‐functional requirements do developers focus
Res. Methods 15(4), 624–662 (2012). https://doi.org/10.1177/ on? an empirical study on stack overflow using topic analysis. In: Pro-
1094428112452760 ceedings of the 12th Working Conference on Mining Software Re-
31. Garousi, V., Felderer, M., Mäntylä, M.V.: The need for multivocal liter- positories, pp. 446–449. IEEE (2015)
ature reviews in software engineering: complementing systematic litera- 45. Lin, B., Serebrenik, A.: Recognizing gender of stack overflow users. In:
ture reviews with grey literature. In: Proceedings of the 20th Proceedings of the 13th Working Conference on Mining Software Re-
International Conference on Evaluation and Assessment in Software positories, pp. 425–429. ACM (2016)
Engineering, pp. 26. ACM (2016) 46. Low, J.F., Svetinovic, D.: Data analysis of social community reputation:
32. Zhang, H., et al.: An evidence‐based inquiry into the use of grey liter- good questions vs. good answers. In: Proceedings of the 22nd Interna-
aturein software engineering. In: Proceedings of the 42nd International tional Conference on Industrial Engineering and Engineering Manage-
Conference on Software Engineering, pp. 1–13. ACM (2020) ment, pp. 1193–1197. IEEE (2015)
33. Modi, V., et al.: Impact evaluation of smoke‐free mass media campaign 47. Riungu‐Kalliosaari, L., et al.: Devops adoption benefits and challenges in
on knowledge, attitude and behavior of the target audience in India. In: practice: a case study. In: International Conference on Product‐Focused
Proceedings of the 2012 National Conference on Health Communication Software Process Improvement, pp. 590–597. Springer (2016)
Marketing and Media (2012) 48. Kitchenham, B., Charters, S.: Guidelines for Performing Systematic
34. Brereton, P., et al.: Lessons from applying the systematic literature review Literature Reviews in Software Engineering. (version 2.3). Technical
process within the software engineering domain. J. Syst. Software 80(4), Report. Keele University and University of Durham (2007)
571–583 (2007). https://doi.org/10.1016/j.jss.2006.07.009 49. Zhou, X., et al.: A map of threats to validity of systematic literature re-
35. Myrbakken, H., Colomo‐Palacios, R.: Devsecops: a multivocal literature views in software engineering. In: 2016 23rd Asia‐Pacific Software En-
review. In: Proceedings of the International Conference on Software gineering Conference (APSEC), pp. 153–160. IEEE (2016)
Process Improvement and Capability Determination, pp. 17–29. Springer 50. Mahood, Q., Van Eerd, D., Irvin, E.: Searching for grey literature for
(2017) systematic reviews: challenges and benefits. Res. Synth. Methods 5(3),
36. Smeds, J., Nybom, K., Porres, I.: Devops: a definition and perceived 221–234 (2014). https://doi.org/10.1002/jrsm.1106
adoption impediments. In: Proceedings of the 2015 International 51. Sammy Migues, J.S., Ware, M.: Building Security in Maturity Model
Conference on Agile Software Development, pp. 166–177. Springer (2019). https://www.bsimm.com/content/dam/bsimm/reports/
(2015) bsimm10.pdf/
37. de França, B.B.N., Jeronimo Junior, H., Travassos, G.H.: Characterizing 52. Fetterman, D.M.: Ethnography: Step‐By‐Step, vol. 17. Sage (2019)
devops by hearing multiple voices. In: Proceedings of the 30th Brazilian 53. Al‐Baik, O., Miller, J.: The kanban approach, between agility and lean-
Symposium on Software Engineering, pp. 53–62. ACM (2016) ness: a systematic review. Empir. Software Eng. 20(6), 1861–1897 (2015).
38. Kao, D.Y.: Performing an apt investigation: using people‐process‐ https://doi.org/10.1007/s10664-014-9340-x
technology‐strategy model in digital triage forensics. In: 2015 IEEE 54. Yasin, A., Hasnain, M.I.: On the Quality of Grey Literature and its Use in
39th Annual Computer Software and Applications Conference, pp. Information Synthesis during Systematic Literature Reviews (2012)
47–52. IEEE (2015) 55. Tyndall, J.: Aacods Checklist. Flinders University (2008)
39. Prodan, M., Prodan, A., Purcarea, A.A.: Three new dimensions to people, 56. Zhang, H., et al.: Ethnographic research in software engineering: a critical
process, technology improvement model. In: New Contributions in In- review and checklist. In: Proceedings of the 2019 27th ACM Joint
formation Systems and Technologies, pp. 481–490. Springer (2015) Meeting on European Software Engineering Conference and Symposium
40. Gill, A.Q., et al.: Devops for information management systems. VINE J. on the Foundations of Software Engineering, pp. 659–670. ACM (2019)
Inf. Knowl. Manag. Syst. 48(1), 122–139 (2018). https://doi.org/10.
1108/vjikms-02-2017-0007
41. Ghaffari, F., Gharaee, H., Arabsorkhi, A.: Cloud security issues based on
people, process and technology model: a survey. In: 2019 5th International How to cite this article: Zhou, X., et al.: Revisit
Conference on Web Research (ICWR), pp. 196–202. IEEE (2019) security in the era of DevOps: an evidence‐based inquiry
42. Bird, J.: DevOpsSec: Delivering Secure Software through Continuous
into DevSecOps industry. IET Soft. 17(4), 435–454
Delivery. O’Reilly Media (2016)
43. Neto, G.T.G., et al.: Multivocal literature reviews in software engineering: (2023). https://doi.org/10.1049/sfw2.12132
preliminary findings from a tertiary study. In: Proceedings of the 13th