Systematic Literature Review

(SLR)
Prof. Dr. M. Haluk Köksal
 Systematic Literature Review
In research, there are different aims and situations in which
scholars write a literature review. Typically, the aim of literature
reviews is to summarize and integrate the existing knowledge about
a topic (Rowley and Slack 2004). The situations where literature
reviews are written and used are independent from the authors
that deal with the topic. In broad terms, there are three main
situations for writing a literature review:
1. A standalone review article of the literature for a specific topic.
2. An introduction to an empirical paper and foundation for
hypotheses.
3. The first stage of a bigger research project (e.g. a dissertation).
 Systematic Literature Review
• A SLR is more useful than a traditional literature
review for a dissertation (especially at doctoral level).
• In empirical articles, on the other hand, a literature
review does not need to answer a research question
on its own; instead, it provides a short overview of the
topic and helps to derive the main hypotheses of the
paper.
• Therefore, a traditional literature review might in
some cases be more suitable for empirical articles,
where the main focus is not the literature review itself.
• Where to start
• How to start
• Where and how to store
• How to use them
• How finding them
• And So many other issues
 Systematic Literature Review
is the key to overcome all these
confusions and difficulties
 Definition of Systematic Literature Review

• Is a scientific study designed to address a specific research

question by comprehensively collecting all the information
available on a topic that is defined at the outset by
absolute inclusion and exclusion criteria.
• Is a systematic way of collecting, critically evaluating,
integrating, and presenting findings from across multiple
research studies on a research question or topic of
interest.
• Is “systematic” since it adopts a consistent, widely
accepted, methodology. The methodology should address
concerns regarding quality issues, such as bias,
replicability, credibility, etc.
 Traditional Review vs. Systematic Literature Review
• Broadly speaking, the two most commonly known types
of literature reviews are the “traditional” or narrative
review and the “gold standard” or SLR.
• Traditional reviews, also known as conventional or
nonsystematic reviews are generally faster and easier to
conduct and are sometimes appropriate due to a short
time frame or lack of resources.
• On the other hand, systematic literature reviews are
more transparent, more replicable and reduces the risk
of bias.
 Systematic Literature Review Traditional Literature Review

Identification for the need for a SLRs only make sense if there is a Traditional reviews are part of
review article need for one. nearly every publication.

Development of a review protocol It is essential for the objectivity of Not common for traditional
a SLR reviews
Identification of It is a structured, replicable and It is a subjective process
research papers transparent process

Evaluating studies It has a transparent protocol for There is no clarity for the
elimination of the studies and the exclusion of the studies. The
elimination process is objective author takes the literature that
helps to support his/her
hypotheses

Conducting data Driven by a general protocol Driven by the intuition of the

extraction authors

Conducting data synthesis Concept driven; central part of an Not necessary; resembles a
SLR summary of existing literature

Reasons for a SLR Standalone paper, creates To set a literature foundation for
evidence and answer a research an upcoming empirical project, as
question individual paper (formerly)
 Steps of Conducting SLR
1. Familiarize yourself with the PRISMA framework.
2. Formulate your clear and unambiguous research question
3. Identify the concepts to be included in you research.
4. Define each term in the study question.
5. Define relevant date sources (i.e. Web of Science or Scopus), typically no more
than five.
6. Develop a series of key words to be used for the search guided by terms that you
defined in step 5.
7. Decide inclusion and exclusion criteria. Define the limits that you will use.
8. Go to each database individually to tailor your key words.
9. Title review.
10. Abstract review
11. Abstract phase screening quality check
12. Critical Review of full articles.
13. Critical Review phase screening quality check
14. Included studies Matrix
15. Writing your report
 Steps of Conducting SLR
1. Familiarize yourself with the PRISMA framework.
PRISMA is an evidence-based minimum set of items for reporting in
systematic reviews and meta-analyses. PRISMA focuses on the
reporting of reviews evaluating randomized trials, but can also be
used as a basis for reporting systematic reviews of other types of
research, particularly evaluations of interventions.
PRISMA framework
 Steps of Conducting SLR
2. Formulate your clear and unambiguous
research question . This
• Allows you to find information quickly.
• Allows you to find information that is relevant
and valid (accurately measures stated
objectives).
• Provides you with a checklist for the main
concepts to be included in your search strategy
 Steps of Conducting SLR
3. Identify the concepts to be included in you
research. Show at least three concepts of interest:
1. population(s) of interest: i.e. the research work
dealing with ecosystem services in mountainous
regions
2. Setting(s) of interest: i.e. the geographical
distribution of existed studies, study distribution
based on categories of mountain ecosystem
services (MES) assessed.
3. Issue(s) of interest: i.e. trends of MES research,
existing knowledge in MES studies, the challenges
and gaps in MES
Geographical spread of the
research
 Steps of Conducting SLR
5. Define each term in the study question.
Do not assume that any term is unequivocally
understood either by the team members or by
the readers.
“Mountain ecosystem” AND “services”
“Mountain ecosystem services”
“Mountain ecosystem” AND “synergies”
“Mountain ecosystem” AND “initiatives”
“Mountain ecosystem” AND “challenges”
 Steps of Conducting SLR
6. Define relevant data sources (i.e. Web of
Science or Scopus), typically no more than five.
It is a good idea to enlist the expertise of a
reference librarian at this point.
 Steps of Conducting SLR
7. Develop a series of key words to be used for
the search guided by terms that you
defined in step 5. Now finalize your plan for
the systematic search. The first sub-step is to
develop a series of key words to be used
for the search.
series of key words
 Steps of Conducting SLR
8. Decide inclusion and exclusion criteria. Define the
limits that you will use.
Terms in the inclusion/exclusion criteria must
also be defined. Define the limits you will use for
your systematic search.
• Decide whether or not you will include gray
literature (non-peer reviewed publications,
dissertations, thesis, conference proceedings, etc.)
and whether or not you will incorporate reference
list searching (snowballing) or hand searches.
 Inclusion and exclusion criteria
Inclusion criteria
• Articles that document and focus on consumer experiences (CX).
• Articles that document business to consumer (B2C) experiences.
• Articles related to the retail experience.
• Articles published between 2010 and 2020.
• Articles written in English.
Exclusion criteria
• Refer to CX in general and are not related to the retail area.
• Document experiences related to business to business (B2B),
consumer to business (C2B) or
• consumer to consumer (C2C).
Phase Inclusion and exclusion criteria

P1 Search based on final search string on major digital libraries to

cover journal articles and conference papers

P2 Excluding unpublished working papers, news articles and non-

English articles

P3 Excluding duplicate reports of the same study (removing

duplicate reports that emerge due to the same search being
performed in different electronic journal databases)

P4 Excluding discussion papers, tutorial and prefaces

P5 Excluding publications that were not truly related to CF

technique and implicit feedback and did not contain the search
strings

P6 Excluding short papers (e.g. poster presentation, summaries of

tutorials) as these papers cannot answer to the research
questions as well

P7 Review of full text papers and excluding studies that are not
related to the research questions
 Steps of Conducting SLR
9. Go to each database individually to tailor your key
words.
When used, they allow the researcher to search
very specifically. This step has an ebb and flow to it as
you develop your search terms and key words. Too
specific and your search will yield nothing, too broad
and your search will yield will be unmanageable. Use
the key word generators within the database to create
a formula of your search terms, using “or,” “and” as
appropriate between terms.
 Database search summary.
Database Search syntax Total Abstract Full text
hıts read downloaded
PsycINFO ‘Big Data Analytics’, 16 16 3
‘Healthcare’
‘Predictive Analytics’, 9 9 0
‘Healthcare’
‘Big Data Analytics’, ‘Health
Management’ 20 20 2

PubMed ‘Big Data Analytics’, 263 263 35

‘Healthcare’
‘Predictive Analytics’, 86 86 12
‘Healthcare’
‘Big Data Analytics’, ‘Health
Management’ 15 15 3

Scopus ‘Big Data Analytics’, 587 587 36

‘Healthcare’
‘Predictive Analytics’, 194 100 4
‘Healthcare’
‘Big Data Analytics’, ‘Health
Management’ 17 17 3

Web of Science
 Steps of Conducting SLR
10. Title review.
Export choose articles to reference
management software such as Zotero, EndNote,
RefWorks, etc.
Then into excel sheet. This is your Journal
Citation Report (JCR).
• JCR includes; Author name, abstract, publication
journal, DOI, volume, year of publication, etc.
 Steps of Conducting SLR
11. Abstract review
• Up to this point you have screened only titles, so now you will
begin to read through the abstracts of each article and screen
them against the exclusion criteria. Instead of inserting “yes” or
“no,” it is helpful to create separate columns for the two and
insert a number “1” (for yes or for no, as applicable) so that you
can run a sum total at the bottom of each spreadsheet to keep
track of how many duplicates were removed, how many were
excluded and how many were included per database.
• Once you have completed this for each separate database, create
a tab for all included articles. Copy and paste all the included
articles into this sheet and sort by title to quickly find and remove
any remaining duplicates. This information will be important data
for the PRIMSA flowchart.
 Steps of Conducting SLR
12. Abstract phase screening quality check
To strengthen validity and reliability for your search
results, work with at least one other researcher to conduct
an inter-rater reliability assessment. For instance, you can give
the second reviewer 20% of the total number of abstracts you
reviewed, randomly chosen, and leave your exclusion fields
blank so that the second reviewer can run a parallel screening
process on the subset of abstracts. Compare your
findings and aim for at least 90% agreement. Discuss any
areas of disagreement with the third team member
reviewer until you reach an inclusion/exclusion agreement.
 Steps of Conducting SLR
13. Critical review of full articles.
Once you have the list of final included
articles, conduct in-depth critical review of each
article. Extract all relevant information from each
article and insert in the appropriate place in the
Excel document. As part of the critical review, rate
the level of quality of evidence for each article
using the framework you have adopted for your
study.
 Steps of Conducting SLR
14. Critical review phase screening quality check
• To strengthen validity and reliability for your critical review
findings, work with at least one other researcher to conduct an
interrater reliability assessment. For instance, you can give the
secondary reviewer 20% of the total number of articles you
critically reviewed, randomly chosen, and leave your
assessment fields blank, so that the second reviewer can run a
parallel critical evaluation process on the subset of articles.
Compare your findings and aim for complete agreement. Closer
the agreement to 100%, more robust is the critical review
process. A level of at least 90% agreement may be acceptable.
Discuss any areas of disagreement with the third team member.
 Steps of Conducting SLR
15. Included studies Matrix

Outcomes Methods

Participants

Year
Level of
Evidence
First
Author
Assessments
Settings Results
 Steps of Conducting SLR
16. Writing your report
 Reporting from SLR
• Not a simple additive task
• Interpret analysis of findings from multiple
studies.
 Reporting from SLR
- Abstract
- Introduction
- Method
- Results
- Discussions
- Conclusion
- Reference
- Appendix
 Reporting from SLR
(Abstract)
• Background, objectives, data sources, study
eligibility criteria, Participants, study
appraisals and synthesis methods, results,
limitations, conclusions and implications of
key findings and inclusion of SLR registration
number if applicable
 Reporting from SLR
(Introduction)
• What is the SLR question?
• What was already known on the
topic/question of interest?
• What drove the need for conducting this
specific SLR?
• What is the objective of this SLR?
• What is the importance of this SLR
study?
 Reporting from SLR
(Method)
• Team composition
How many investigators were on the team?
What were their qualifications, expertise, and
levels of experience?
 Reporting from SLR
(Method)
• Search preparation
What databases were targeted? What key
words/phrases were used in each database
search? What were the inclusion and exclusion
criteria?
 Reporting from SLR
(Method)
• Search Process
- How and when were the database searches
conducted? How many phases of search were
involved?
- How we exclude/include specific publications?
- What was involved in each steps
 Reporting from SLR
(Method)
• Critical Review
- How it was conducted?
- What information was extracted?
- The quality level of each study determined
- Used framework and method
 Reporting from SLR
(Result)
• Description of final list
- How many articles were find in each step?
- How many were retained in each step?
- The final number of articles the critical review
were performed
 Reporting from SLR
(Result)
• Content of final list
- How many articles were empirical/rational an
experimental?
- How many were qualitative/quantitative?
- What were the key finding?
- The level of quality of each articles
 Reporting from SLR
(Discussion)
- What new information was generated from this
systematic review?
- What is the general level (quality and magnitude)
of evidence in various thematic areas?
- Which of the areas enjoy higher quality/or
magnitude of evidence?
- Which areas currently lack quality and/or
magnitude of evidence?
 Reporting from SLR
(Limitations and future studies)
- What are the limitations of this SLR?
- What topics/questions should be examine in future
studies?
 Reporting from SLR
(Conclusion)
- What are the key messages of this SLR for your target
readers?
(Reference)

(Appendix)
 References
• Booth, Sutton, & Papaioannou, (2016), Systematic Approaches to a Successful
Literature Review. Sage , London . ISBN 9781473912465
• Guillaume, L. (2019), Systematic literature reviews: an introduction, International
Conference On Engineering Design, Iced19, 5-8 August, Delft, The Netherlands
• Huelin, R., Iheanacho, I., Payne, K. & Sandman, K. What’s in a name? systematic
and non-systematic literature reviews, and why the distinction matters,
https://www.evidera.com/wp-content/uploads/2015/06/Whats-in-a-Name-Syst
ematic-and-Non-Systematic-Literature-Reviews
and Why-the Distinction Matters.pdf
• Kraus, S., Breier, M. & Dasí-Rodríguez, S. (2020), The art of crafting a systematic
literature review in entrepreneurship research, Int. Entrep. Manag.
J., 16(3), 1023–1042.
• Pati, D. & Lorusso, L. N. (2018), How to write a systematic review of the
literature, Health Environments Research, & Design Journal, 11(1), 15-30.
• Robinson, P. & Lowe, J. (2015), Literature reviews vs systematic reviews,
Australian and New Zealand Journal of Public Health, 39(2), 102-103.
• Xiao, Y. & Watson, M. (2019), Guidance on conducting a systematic literature
review, Journal of Planning Education and Research, 39(1), 93-112