See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/303703556

Search Strategy Formulation: A Framework For Learning

Conference Paper · June 2016

DOI: 10.1145/2934732.2934752

CITATIONS READS
3 4,845

2 authors:

Andrew Macfarlane Tony Russell-Rose

City, University of London 2Dsearch
100 PUBLICATIONS 1,167 CITATIONS 116 PUBLICATIONS 3,372 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Designing the Search Experience (book) View project

Oracle/Endeca View project

All content following this page was uploaded by Tony Russell-Rose on 25 January 2018.

The user has requested enhancement of the downloaded file.

 Search Strategy Formulation: A Framework For Learning
Dr Andrew MacFarlane Dr Tony Russell-Rose
Department of Computer Science UXLabs
City University London 3000 Cathedral Hill
London EC1V 0HB Guildford, GU2 7YB
+44 (0)20 7040 8386 +44 7779 936191
[email protected] [email protected]

ABSTRACT strategy development and provide a framework for learning based

Healthcare information professionals perform systematic literature on this methodology. In section 2 we provide the background to
reviews to gather the evidence needed to answer specific research the problem in the context of the domain, and then discuss related
questions and formulate policy. However, performing a pedagogical work in section 3. Section 4 details the structured
systematic review is a resource-intensive and time consuming searching methodology which we then apply in Section 5 to
undertaking, often taking years to complete. Moreover, the output develop a framework for learning. We provide ideas for the future
relies heavily on the quality of the initial search strategy in direction of the work in section 6.
ensuring that the scope is sufficiently exhaustive and not biased
by easily accessible studies. In this paper we introduce a
2. BACKGROUND
structured methodology and a framework for learning which At its heart, the process of systematic review relies on painstaking
together aim to embody best practices from the community and and meticulous searching of multiple literature sources. These
provide support for many of the common issues in search strategy include published literature sources such as MEDLINE and other
development. specialist databases and ‘grey literature’ (i.e. technical reports and
other non-peer reviewed sources). The principal way in which
CCS Concepts such sources are interrogated is through the use of Boolean
• Information systems➝Information Retrieval➝Information queries, which utilize a variety of keywords, operators and
Retrieval Query Processing. ontology terms (also referred to as ‘subject headings’) – see Fig.
1.
Keywords ("etiology"[Subheading] OR
Information retrieval; systematic reviews; education; training. "etiology"[All Fields] OR
"causes"[All Fields] OR
1. INTRODUCTION "causality"[MeSH Terms] OR
Medical knowledge is growing so rapidly that it is difficult for "causality"[All Fields]) AND
healthcare professionals to keep up. As the volume of published ("somnambulism"[MeSH Terms] OR
studies increases year by year, the gap between research "somnambulism"[All Fields] OR
knowledge and professional practice grows ever wider. Systematic ("sleep"[All Fields] AND
literature reviews can play a key role in closing this gap, by "walking"[All Fields]) OR "sleep
synthesizing the complex, incomplete and at times conflicting walking"[All Fields])
findings of biomedical research into a form that can readily
Fig. 1 – Example of subject headings use
inform health decision making [1]. A key principle of systematic
reviews is that the protocol by which the literature was collected Reviewers incrementally build complex queries line by line,
and analyzed should be made transparent and repeatable. sometimes involving hundreds of terms, which are combined to
form an overall search strategy – see Fig. 2.
However, undertaking a systematic review is a resource-intensive
and time consuming process, sometimes taking years to complete The choice of search strategy is critical in ensuring that the
[2]. Even rapid evidence assessments, designed to provide quick process is sufficiently exhaustive and that the review is not biased
summaries of what is known about a topic or intervention, can by easily accessible studies [5]. In addition, the strategy needs to
take as long as two to six months [3]. Moreover, new research be transparent and repeatable, so that others may replicate the
findings may be published in the interim [4], leading to a lack of methodology. However, there are often mistakes in search
currency and potential for inaccuracy. It is therefore vital that the strategies reported in the literature that prevent them from being
search strategies used to identify relevant studies should be executed in their published form. In one sample of 63 MEDLINE
published so that the process is seen to be auditable and strategies, at least one error was detected in 90% of these,
repeatable. including spelling errors, truncation errors, logical operator error,
incorrect query line references, redundancy without rationale, and
In this paper we introduce a structured methodology for search more [6].
SAMPLE: Permission to make digital or hard copies of all or part of this
Evidently, despite the dedication and painstaking attention to
work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage detail of many individuals (many of whom are trained librarians),
and that copies bear this notice and the full citation on the first page. To creating effective search strategies may be prone to error, often
copy otherwise, or republish, to post on servers or to redistribute to lists, relying on manual processes with limited editorial support.
requires prior specific permission and/or a fee. Moreover, once published, strategies are typically stored as free
CERI’16, June 14–16, 2016, Granada, Spain. text, and are thus rarely directly executable in their native form.
Copyright 2016 ACM 1-58113-000-0/00/0010 …$15.00. This compromises their ability to be used by others and may
DOI: http://dx.doi.org/10.1145/12345.67890 unintentionally detract from the rigour of the review.
 1. randomized controlled trial.pt. with is whether to focus on theory or practice in teaching [7]: the
2. controlled clinical trial.pt. argument for the former is that understanding the theoretical
3. randomized.ab. concepts provides the student with transferable skills (i.e. they can
4. placebo.ab. use any system to search). However systems do vary in practice
5. clinical trials as topic.sh. and it is important to give the student practical knowledge of
6. randomly.ab. specific systems. In this work, we recommend a balance of the
7. trial.ti. two.
8. 1 or 2 or 3 or 4 or 5 or 6 or 7
The Cochrane Organization provides guidelines for search
9. (animals not (humans and
animals)).sh. strategy development and has developed curricula specifically for
10. 8 not 9 search in systematic review [2]. Professional bodies such as CILIP
11. exp Child/ in the U.K and the Medical Library Association in the U.S.A. [11]
12. ADOLESCENT/ also provide guidelines for curricula design; of a more general
13. exp infant/ nature in the former case and more specific in the latter case.
14. child hospitalized/ Other organizations such as the UK Quality Assurance Agency
15. adolescent hospitalized/ provide subject benchmarks in Library and Information Science
16. (child$ or infant$ or toddler$ or which can be used to inform curricula.
adolescen$ or teenage$).tw. After a curriculum has been defined, the instruction method can
17. or/11-16 be derived and there is a clear case for producing materials online
18. Child Nutrition Sciences/ through E-learning systems such as Moodle. Much work has been
19. exp Dietary Proteins/ done in the area of developing online resources for IR instruction
20. Dietary Supplements/ [7]; for example, the Cochrane Organization materials are online
21. Dietetics/ and available to all [2]. E-Learning environments may include
22. or/18-21 interactive elements to help the student e.g. Java Applets [12], or a
23. exp Infant, Newborn/ self-training package which can include pre and post self-
24. exp Overweight/ assessments for students to measure their own progress [13].
25. exp Eating Disorders/ Materials and assessments are designed in conjunction with each
26. Athletes/ other and are integrated to ensure a good learning experience for
27. exp Sports/ the student. Face to face methods of delivery can also be used [7].
28. exp Pregnancy/
29. exp Viruses/ Assessment and feedback can use a wide variety of methods
30. (newborn$ or obes$ or "eating including those mentioned above in E-learning such as multiple
disorder$" or pregnan$ or childbirth choice questions (MCQs) [14], assignments, projects and tests
or virus$ or influenza).tw. [15] or even an IR game [16] where the student can assess their
31. or/23-30 performance on a task using an IR system with a standard test
32. 10 and 17 and 22 collection, allowing them to assess their progress using a
33. 32 not 31 graphical tool. Feedback can be automatic via MCQs [14] or via
summative or formative textual feedback [15] for more abstract
Fig. 2 – Complex Boolean query example extract ideas where there is no right and wrong answer (see section 5
below).
3. RELATED WORK
Students of library and information science may have taken a 4. A STRUCTURED SEARCH
module on search strategy development in library school, but
further education and training is required to deal effectively with METHODOLOGY
the highly complex queries typical of systematic reviews. We When undertaking a search there are a number of stages in the
provide a broad overview of teaching methods, curricula process starting from the realization of an information need
development, online materials and assessment and feedback (cognition) to the creation of a search which is submitted to an
below. A much larger review of the area can be found in information retrieval system (syntactic). At City University
Fernandez-Luna et al. [7] – here we focus on Level 2 of the London a search methodology has been used for many years on
taxonomy given in that paper, together with a focus on [B] various modules which provides a structured approach to the
Educational Goals; [b1] Library and information science. The process from beginning to end. The model resembles the
technical level [A] focuses on the operational aspect of framework derived by Taylor in 1968 [17], but is much more
undertaking a query from a given information need. elaborate. We outline each of the levels of the structured
searching framework in this section.
The literature on teaching methods shows that generic ideas in
pedagogy can be used to build frameworks to tackle problems in 4.1 Cognitive stage
teaching and learning in information retrieval [7]. There are a This stage initiates a search. The user realizes that they do not
couple of tensions which need to be considered. The first of these have sufficient knowledge to undertake a particular task [18], or
is that the method can either be process or outcome based. in the case of systematic review is faced with a complex
Kuhlthau [8] provides a five stage strategy to assist the process of information need in the form of a specific research question. The
learning based on prior work [9]. These kinds of ideas are very search process is often performed by an intermediary (e.g. a
useful for systemic reviews, where the search process is very librarian) who may have limited knowledge of the subject area.
complex. McGregor [10] however shows that students tend to be Therefore an ability to understand the needs of the original
focused on the outcome rather than the process, so any scheme for requestor is key at this stage. Needs can be visceral (an
education must deal with that tension and ensure the student unexpressed need) or conscious (a within brain description) [17].
understands the importance of process. The second tension to deal For the most part, systematic review needs are conscious, as
domain knowledge may be required. Clarity of self-reflection is 4.3.3 Citation Pearl Growing
essential here as initial thoughts about the research question will In this method a known useful item is pre-identified and index
govern the direction of the search process, and further reflection terms and or subject headings are extracted from it. The user goes
during iterations of searching will assist in understanding the topic through several iterations of extracting terms from records and
and further guiding the direction of the search. testing them out on queries until they are happy with both the
terms and their combinations [21]. A final phase is to reuse the
4.2 Linguistic stage building blocks strategy to create the final set of results.
Once the searcher has identified an area or sub-domain in which
to undertake a search, some kind of linguistic description is 4.4 Tactical stage
required to identify the underlying concepts. This could take the Within the strategies outlined above a number of tactics are
form of a document which describes the overall need, or it could available to the user. These fall into two broad groups: choice of
be a description of facets using schemes such as PICO – Patient, terms and choice of operators. Choice of terms relates to the
Intervention, Comparison and Outcome [19]. Often it may involve searcher’s domain knowledge, whilst choice of operators relates
both with the facets defined in the document describing the to their knowledge of search techniques. The choice of terms can
information need. Other more generic facet analysis schemes be augmented by the use of field operators, which depends on the
include ad-hoc and PMEST – Personality, Matter, Energy, Space meta-data available (e.g. restricting a search to the AUTHOR
and Time [20]. Whichever scheme is used, for each facet a list of field). Operators can be either Boolean (AND, OR and AND-
terms and synonyms is identified. NOT are the only operators available on most systems) or word
based. The latter can be either proximity operators (e.g. for
4.3 Strategic stage phrases: “sleep walking” in Figs. 3/4) or truncation/wildcard
Assuming Boolean logic is used, the general scheme for taking operators (e.g. caus* in Figs. 3/4). While word based operators are
data from the facet analysis is to apply an OR operator to the not strictly Boolean operators, they behave in a similar manner,
terms and synonyms within a facet, and then to apply the AND e.g. by narrowing a query (proximity operators are a special case
operator between the facets. There are three well-known search of AND) or broadening a query (truncation/wildcard operators are
strategies for combining terms and facets: building blocks, a special case of OR).
successive fractions and citation pearl growing [21]. We outline
these schemes here, and assume access to intermediate search sets 4.5 Logical stage
as in the examples given in Figs. 1 and 2. A search strategy and its tactics are formed within some kind of
4.3.1 Building Blocks logical framework – in the examples used so far we have
In this method the sets for the facets are formed separately, and concentrated on Boolean logic and word based extensions to that
once this is done the final set is formed using the AND operator logic. This type of search is exact match logic using set retrieval,
on the facet sets (see Fig. 3). which is the dominant paradigm in systematic reviews. Another
form of logic is best match in which ranked retrieval is addressed,
Set1 = etiology OR caus* but this is rarely relied upon exclusively in systematic reviews.
Set2 = somnambulism OR sleep We address exact match logic only in this paper therefore.
Set3 = walking OR "sleep walking” Searching requires the users to utilize their knowledge of strategy,
Set4 = Set1 AND Set2 AND Set3 tactics and logic all together – which form the formalized need
Fig. 3 – Building Blocks Search Strategy example [17]. However logic can be considered separately in terms of
learning, as we will outline later in section 4.
The advantage of this approach is that each set (sets 1 to 3 in Fig.
3) can be reused, and the search can continue to develop themes 4.6 Syntactic stage
within each facet with no impact on other facets. However the This is the stage where the user takes their search strategy and
drawback of this approach is that the search may lose focus in executes it on an operational information retrieval system – the
terms of its overall direction which may undermine its compromised need (question as presented to the information
effectiveness. system) [17]. Each IR system will have its own syntax, and the
4.3.2 Successive Fractions types of operators available and their range will vary from system
An alternative to building blocks is successive fractions, where to system. In most cases AND and OR are used as is, but most
one facet is formed first and subsequent facets are incrementally systems use NOT to mean AND-NOT (see set 10 in Fig. 2) which
added to the set to form the final answer (see Fig. 4). can be confusing since the Boolean operator NOT is actually a
unary rather than binary operator.
Set1 = etiology OR caus*
Set2 = Set1 AND (somnambulism OR sleep) Many systems used ‘*’ and/or ‘?’ for wildcard characters either
Set3 = Set2 AND (walking OR "sleep for single characters, a given set of characters or any number of
walking”) characters to an upper limit. Proximity operators often use quotes
Fig. 4 – Successive Fractions Search Strategy example “” (as per Google), but many offer the ability to choose the
number of characters between the terms e.g. WITHIN, NEAR.
The searcher can start with the most general facet and refine the Proximity operators on blocks of text such as sentences or
query from there. There are fewer steps in this method and it is paragraphs are rare. A further syntactic method used often is to
more holistic, with the searcher having a clearer idea in each restrict the search to a given field (e.g.
iteration of where the search is going. The drawback of the “etiology"[Subheading]) which vary syntactically
method is that any mistakes in earlier steps (e.g. Set3 in Fig 4), between systems (e.g. AU(name) in ProQuest Dialog [22]). The
may require the search process to be restarted (e.g. from Set1 in meta-data scheme available on the IR system or the source will
Fig. 4). determine the fields available.
Finally the type of interface will often determine the type of 5.1.3 Teaching methods
strategy used. The search strategies described in section 4.3 Clearly the student needs to take a step back and understand the
require access to search sets – a typical example of this is information need in detail before attempting searching as
ProQuest Dialog [22]. Alternatively, in some cases a form-based recommended by Cohen [28]. The first author has used this
user interface is available with columns representing terms, method of a number of years where information needs are taught
synonyms and phrases and rows representing the facets. In this in conjunction with the linguistic stage, and this allows the student
paper we concentrate on the former, known as command line to focus on getting things right at the start. Putting students in
interfaces. groups and getting them to discuss the issues in tutorials has
found to be a very successful form of learning [29].
5. A FRAMEWORK FOR LEARNING
In this section we develop a framework for learning based on the 5.1.4 Assessment and Feedback methods
methodology in section 4. Systematic review often requires two A focus on assessment could involve encouraging students to
main activities [5]: an initial search to identify any existing develop their self-reflection skills through either formative or
systematic reviews on the subject, and a full search if/when no summative feedback schemes (perhaps even using peer review).
prior review is found. The framework can be applied in both Assessment would focus on information literacy tasks, assessing
activities, but is more critical for the latter due to the complexity the information quality of sources using knowledge positive and
of the task. At each stage we outline good practice and identify negative examples and their relation to relevance.
common sources of error [6], outline learning objectives, curricula
and learning materials, teaching methods and assessment and 5.2 Linguistic stage
feedback methods. A key issue at this stage is the ability to define an appropriate
research question [3] given the information need identified at the
5.1 Cognitive stage cognitive stage. It would be useful to practice the writing of a
A key problem at this stage is the growing volume of published document which describes this question, including its objectives,
medical studies [1],[4] with the number increasing year on year. It the subject area, the population concerned, type of evidence for
is worth stressing the need for the user to reflect on the current evaluation and outcomes required [3]. Once this is done, support
state of their search and its ability to identify relevant studies that can be provided for the extraction of facets from the healthcare
address the research question. Searchers should also be aware of question, using PICO [19] or some other appropriate facet
the importance of sources, i.e. the databases that contain relevant analysis scheme. If appropriate or available, the use of tools for
information to fulfill the given information need. This will include extracting PICO elements or other information using utilities such
peer review literature in prestigious journals, but other sources as ExaCT [5] could be useful. Section A of the standard checklist
should be included such as non-English language articles, the [24] can be used to collect information about the information need
‘grey’ literature, non-refereed journals, conference proceedings, including the authors’ stated objective, the focus of the research
company reports etc. [3]. This ensures that the searcher etc.
understands the comprehensive nature of the requirements of a
systematic review, and that the search needs to be exhaustive
5.2.1 Learning Objectives
The learning objectives could include the following: 1) defining
before any filtering of the literature can take place [3]. However
and documenting a clear research question; 2) Using facet
searchers should be aware of information quality given the range
analysis techniques such as PICO to analyze the research question
and type of material available e.g. potential bias or error in
effectively.
published studies. To this end it would be useful to introduce the
searcher to information literacy ideas to think through these issues 5.2.2 Curricula and learning materials
[23]. Standard checklists are used by search professionals to Reference to good practice provided by Cochrane [2] on how to
validate their search strategy [24]. This includes the identification conduct systematic reviews would be appropriate. This would
of a gold standard of known relevant records in section B of the give the student an overall idea of how to initiate work on a
standard checklist [24] which can be used to further citation pearl research question and keep it up to date. A collection of example
growing search strategies. facet analyses for healthcare topics (e.g. with the PICO scheme)
5.1.1 Learning Objectives would be useful learning materials.
The learning objectives could include an understanding the 5.2.3 Teaching methods
following: 1) the importance of sources and potential bias in those Since facet analysis is not an exact science, students should be
sources, assessing information quality; 2) the exhaustive nature of encouraged to develop their own ideas and to refer to case studies
systematic reviews and the process as a whole; 3) the notion of and examples illustrating good practice. This can be done
relevance and the use of gold standard records to assist strategy individually or in group tutorials, or through online tasks using E-
development. learning materials.
5.1.2 Curricula and learning materials 5.2.4 Assessment and Feedback methods
A key resource would be the Cochrane Handbook for Systematic As with the previous level, both formative and summative
Reviews of Interventions [2], but general schemes on the body of feedback schemes are appropriate, providing individualized
professional knowledge from organizations such as CILIP would feedback to address specific student issues. The use of MCQs
also be useful. There is useful work by Bates et al. [26] which could be considered, but only to address known issues in facet
surveys LIS curricula in Europe and recommends the use of analysis such as placing the terms in the correct PICO element.
Wilson’s nested model [27] to guide curricula design – this would
embed the learning materials in the key academic work on 5.3 Strategic stage
information seeking over many years. This stage is concerned with translating the facet analysis to the
search strategy. A key issue is understanding the relationships
between facets: in the case of Boolean search strategies, OR is
typically applied to terms within facets, and the AND operator is that all ‘pearls’ are retrieved. The balance of the two can be
applied between facets (see section 4.3). Students can confuse the adapted to the given needs of the searcher, but the linkage
two and use inappropriate operators e.g. AND within a facet. One between the different terms needs to be emphasized. Choice of
author has been teaching this material for 15 years, and it is a further strategies such as building blocks can then be addressed.
common source of error.
5.3.4 Learning Objectives
A number of key problems at the strategic level are identified by The learning objectives could include the following: 1) effective
Sampson et al [6]. In some cases the wrong line number is used in translation of facet analysis into an appropriate research strategy;
a step, either omitting a set or using an incorrect set (this applies 2) understanding the different forms of search strategy, their
to any of the strategies described in section 4.3.1 to 4.3.2). The similarities and differences and when to apply a given strategy for
searcher can avoid this by drawing the relationship between line a particular problem.
numbers/sets, to show the relationship between or within facets
depending on the focus. MeSH and free text terms used on the 5.3.5 Curricula and learning materials
same line can compromise reuse. A simple solution to this is to The curricula would focus on the different forms of search
address the granularity of the strategy, and provide examples of strategies available, with a clear link made to the facet analysis.
when MeSH and free text terms could be decoupled. Terms can be The problems identified early in this section should be specifically
reused leading to redundancy without rationale, which may not addressed and built in to the learning materials. Each of the search
harm the search but may slow down run times for large searches strategies needs to be clearly explained with appropriate
and complicate the strategy unnecessarily. A way round this is to examples, with differences between building blocks, successive
check for the use of a given term more than once in a strategy, and fractions and citation pearl growing demonstrated.
ensure that the term is required at that particular stage. In the case 5.3.6 Teaching methods
where searches are required over a number of databases, training
There are a number of different methods for teaching search
on how to tailor the search strategy to each database should be
strategies including Bhavnani et al [30], which uses taxonomies of
provided. This should include a clear description of the strategy
both tasks and general IR strategies to build a methodology to
for the purposes of reproducibility (which is good practice in
learn to search by 1) learning specific search strategies for
systematic reviews). Section C of the standard checklist [24]
frequent tasks, 2) using strategies for given contexts, 3) learning
provides examples of issues to think about when forming the
how to execute a strategy accurately and 4) applying strategies
search strategy, including adapting an already existing search
across different applications (in conjunction with the syntactic
strategy, using a database thesaurus and thinking about how the
level below). Use of graphical online tools would also be a useful
final combination of terms were selected (see sections 5.3.1 to
addition to the learning experience e.g. the relations between
5.3.3 below).
intermediate search sets.
5.3.1 Building blocks 5.3.7 Assessment and Feedback methods
A key issue with this strategy is to get users to understand the
The use of MCQs can be used to test understanding of the form of
drawbacks of the method e.g. a searcher focusing on one facet
strategy, e.g. MacFarlane [14] specifies an example set of
may lose focus on the whole topic (section 4.3.1). Users should be
questions (labelled under the group C element of the MATH
trained to understand that if they are to use the method, a clear
taxonomy [31]), which would use questions on the different forms
understanding of each facet must be gained. This could include
to allow the user to assess their own understanding. For example,
continual review of the information need and any related topics giving the student a facet analysis and asking them to identify the
which could be useful for each facet. Links and relations between
correct building blocks strategy from a number of distractors. Key
facets should be identified by the searcher and recorded in a
problems identified in the Common errors should be built in to the
checklist [14].
distractors, e.g. using OR between facets instead of AND.
5.3.2 Successive fractions
In this strategy the sequential order of the facets is crucial, and the 5.4 Tactical stage
user needs to be taught to think about the starting point. Normally The strategic and tactical stages are closely related and often need
this would be the most specific facet first (e.g. the type of patient to be considered simultaneously. This requires thought on the use
in PICO), with other more general facets following after (e.g. of terms and operators (section 3.4).
outcome in PICO). This is particularly important in more ad-hoc A number of common errors at the tactical level are identified by
methods of analysis where the user has identified their own facets Sampson et al. [6]. Spelling errors are a significant issue.
e.g. Object, Activity, Date. In such cases it would be better to start Applying appropriate thesauri or other knowledge organization
with the Object/Activity facets and finish with the Date facet. As schemes (e.g. taxonomies, ontologies) can require further
with building blocks links and relations between facets should be verification of medical terms. Google may be used as a source of
identified by the searcher and recorded in a checklist [24]. verification but has limited value as the terms returned may reflect
5.3.3 Citation Pearl Growing similar errors made on the web and may not provide relevant
terms for the domain. Missed spelling variants can be dealt with
This requires an understanding of the use of gold standard records
by teaching the searcher to think about variations of words and
(the ‘pearls’) to develop an overall strategy. Section D of the
use truncation as a tactic. However, the searcher can inadvertently
checklist [24] provides useful advice on considering issues such
choose irrelevant MeSH or free text terms, or alternatively miss
as sensitivity (recall), precision and specificity [25]. The ‘pearls’
other useful MeSH terms. A further problem is that MeSH terms
can be used to check each metric and the strategy developed to
can be exploded without any effect if the term is at the bottom of
meet a certain criteria e.g. a preference for a high level of
the hierarchy, since no further child terms exist. Encouraging the
sensitivity (recall) whilst ensuring a threshold of 50% for
learner to reflect on the terms used and providing training on the
specificity [25]. This is done by checking to see if the ‘pearls’ are
MeSH scheme can help address these issues.
retrieved by the search strategy, and an interactive process in
search strategy development may be needed to in order to ensure
Section C of the standard checklist [24] provides examples of wildcards can then be further explained from a Boolean logic
issues to think about when forming tactics, including terms perspective.
extracted from documents and identifying different types of term
checking including terms extracted from gold standard records,
5.5.3 Teaching methods
terms suggested by experts and from database thesauri etc. Online delivery of the material would be appropriate for this level,
with examples and self-assessment for each of the operators. The
5.4.1 Learning Objectives teaching scheme must not assume that the student is familiar with
The learning objectives could include the following: 1) how to discrete mathematics [29]. Tutorial group tasks have also proved
successfully use appropriate tactics within a given strategy, 2) to be successful for face to face students [29].
good practice on choosing operators, 3) good practice on choosing
terms.
5.5.4 Assessment and Feedback methods
Group A questions [31] could be used to assess the understanding
5.4.2 Curricula and learning materials of Boolean and extended Boolean logic by providing text
The learning materials would focus on when to use particular examples and asking the student to pick which queries would
operators in a strategy, e.g. Boolean, proximity or wildcard retrieve that text [14].
operators, and best practice on picking terms e.g. those extracted
from gold standard records. 5.6 Syntactic stage
Implementing the search strategy on an operational information
5.4.3 Teaching methods retrieval system is the final stage of the search (section 4.6). The
Given the subjective nature of term selection, students can be put syntax of the different search systems can be very different but
into groups and given case studies along with examples of good there are certain commonalities. In cases where multiple searches
and bad tactics for those strategies. The use of operators is more are required, training on translation of queries to different systems
objective, and online self-reflection materials can be used. should be provided. This includes training on unary operators
5.4.4 Assessment and Feedback methods (applied to a single term), binary operators (applied to two terms)
For term selection tactics, either formative or summative feedback and clarification of what operators are symmetric (two different
schemes would be appropriate, providing individualized feedback terms can be on either side of the AND, OR operators) and non-
to address specific student issues. MCQs can be used for operator symmetric (in Dialog ProQuest [22] the proximity operators
tactics with appropriate use with given distractors, which can be “”/PRE impose order on words, whilst NEAR does not).
delivered with Group C questions [31] in strategies above [14] but One particular problem at the syntactic level is identified by
as a separate question set, e.g. the correct use of MeSH terms. Sampson et al. [6]. This is the inappropriate use of truncation e.g.
using methods* instead of ‘method*’ to capture several terms on
5.5 Logical stage that concept. Training on truncation operators and their impact
Closely aligned with the tactical stage is the logical stage of the needs to be provided and examples given of both appropriate and
framework (section 3.4). Two key problems at the tactical stage inappropriate use.
are identified by Sampson et al [6]. The first of these is confusion
between the operators AND, OR with potential serious impact to
5.6.1 Learning Objectives
the overall search strategy (section 3.4). This can occur with users The learning objectives could include the following: 1)
unfamiliar with Boolean logic who are used to thinking in terms understanding how to translate a Boolean search strategy with
of AND as an OR: for example a request such as ‘Find me relevant tactics into a form which can be executed by an
documents about cats and dogs’ is linguistically AND, but operational information retrieval system.
semantically it implies OR. This contrast can be confusing for 5.6.2 Curricula and learning materials
students. Clarification on the natural language use of OR and Materials will need to be developed for specific systems e.g.
AND needs to be highlighted to the user. The second issue is the ProQuest Dialog [22], together with a general scheme of how to
inappropriate use of the NOT operator, which must be used with approach the translation of a generic Boolean query to relevant
care as relevant documents may be eliminated from results. It syntax. This will require a survey of existing systems used in
should be stressed to the learner that the NOT operator should systematic review. The material will need to address problems
only be used where a given term or set of terms is known to be identified in the literature mentioned above [6].
harmful to the overall search. Further training could be given on
the relationship between the word operators (truncation, 5.6.3 Teaching methods
proximity) and Boolean operators (OR, AND) ensuring they At this stage practice on real systems will be required to ensure
understand that the former are special cases of the latter (section that the user can truly understand the final stage. This could
4.5). require the use of PC labs, with specific tasks – perhaps in
conjunction with an overall task from all levels of the framework
5.5.1 Learning Objectives – with work on other levels being done prior to the lab. The
The learning objectives could include the following: 1) correct use teaching method needs to instill some self-reflection, to establish
of Boolean and extended Boolean operators. both the process of translation of the Boolean query to the target
5.5.2 Curricula and learning materials system, but also to instill confidence in the student in what can be
The material would focus on understanding Boolean logic using a very complex activity. Online materials and self-assessments on
methods such as Venn diagrams, together with providing some individual elements of the system syntax would also be useful.
understanding of the underlying axioms of the mathematics e.g. 5.6.4 Assessment and Feedback methods
AND, OR are symmetric, whereas NOT is not symmetric. This Assignments which give the student an opportunity to build their
material can be drawn from any good textbook on discrete confidence and knowledge in search e.g. providing an example
mathematics. The use of word operations e.g. proximity and systematic review case study to search for and allowing them to
build an operational query to find information for that case study.
In-class tests could also be used, whereby students are provided share their knowledge with others who have common interests. A
pre-defined search strategies and given limited time to form real broader outcome is to improve the quality of search strategies
searches using a given system in a lab. Multiple choice questions used in systematic reviews, thereby improving the quality and
can be used to tackle Group B questions, focusing on specific accuracy of those reviews.
issues or known problem with syntax on a given search service
[14]. An example would be to give a list of search forms in the 7. ACKNOWLEDGEMENT
given syntax and get the student to choose the number of correct The authors are grateful to Stephen Robertson, Lyn Robinson and
forms [14]. David Bawden for their feedback on the original of the structured
searching framework, and to Stephen Robertson for
recommending the Taylor 1968 reference [7].
6. SUMMARY AND CONCLUSION
We have introduced here a structured search methodology which 8. REFERENCES
is used to inform a framework for learning how to develop search [1] Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J.
strategies which can be used in systematic reviews. This P. T., Mavergames, C., and Gruen, R. L. 2014. Living
framework includes a number of discrete but interlinked stages: systematic reviews: an emerging opportunity to narrow the
cognitive, linguistic, strategic, tactical, logical and syntactic. The evidence-practice gap. In: Plos medicine, Vol. 11, No. 2.
learning framework applied to each stage is a follows:
[2] Lefebvre, C., Manheimer, E., and Glanville, J. 2011.
Cognitive: In this stage the importance of assessing sources will Searching for Studies. In Higgins, J. P. T., and Green S., Eds.
be stressed, in particular understanding the issue of information Cochrane Handbook for Systematic Reviews of Interventions
quality and potential bias in publications. Ideas and concepts in Version 5.1.0 [updated March 2011]. The Cochrane
information literacy can be used to inform this part of the Collaboration. Available on: http://handbook.cochrane.org/.
framework.
[3] Hemingway, P. and Brereton, N. 2009. What is a systematic
Linguistic: A key skill here is forming a research question given a review? 2nd ed. Hayward Medical Communications.
clinical need, and using an appropriate facet analysis scheme to [4] Shojania, K.G., Sampson, M., Ansari, M,T,, Ji, J., Doucette,
identify the complementary concepts of the need. Training in the S. and Moher, D. 2007. How Quickly Do Systematic
use of standard facet analysis schemes such as PICO are required, Reviews Go Out of Date? A Survival Analysis. Ann Intern
together with training on software which can be used to build the Med.
facets.
[5] Tsafnat, G., Glasziou, P., Choong, M.K., Dunn, A, Galgani,
Strategic: Being able to take the facet analysis and form an F. and Coiera, E. 2014. Systematic review automation
appropriate search strategy is the key skill that needs to be technologies. Syst Rev 3,74, 1-15.
developed at this stage. This includes in initial translation from
[6] Sampson M, McGowan J. 2006. Errors in search strategies
the facet analysis to the strategy (OR is applied within facets,
were identified by type and frequency. J. Clinical
AND between facets), to choosing the type of strategy to be used:
Epidemiology. 59, 10, 1057–63.
building blocks, successive fractions or citation pearl growing.
[7] Fernandez-Luna J.M., Huete, J.M. MacFarlane, A. and
Tactical: With a strategy, the choice of terms and operators needs Efthimiadis, E.N. 2009. Teaching and learning in
to be considered. Choice of terms will depend on domain information retrieval. Inf Retrieval, 12, 201-226.
knowledge and interaction with a subject matter expert, whilst
choice of operator requires the appropriate knowledge of Boolean [8] Kuhlthau. C.C. 1997. Learning in digital libraries: an
operators and proximity operators that extend Boolean logic in information search process approach. Lib Trends 45,4, 708-
various ways. Training on the use of field operators would also be 724
appropriate. [9] Kuhlthau, C.C. 1988. Developing a model of the library
search process: cognitive and affective aspects. RQ, 232-242.
Logical: An understanding of the operators identified in the
tactical stage is required, in particular the differences and [10] McGregor, J. 1994. Information seeking and use: students’
relationships between the operators need to be established as well thinking and their mental models. J. Youth Services Lib, 8,
as the appropriate use of operators. 69-76.
Syntactic: This final stage needs to be carried out with an [11] Nicholson, S. 2005. Understanding the foundation: the state
operational information retrieval system, and an understanding of of generalist search education in library schools as related to
the systems functionality must be provided. The system’s ability the needs of expert searchers in medical libraries. J. Med.
to handle intermediate search sets must also be stressed to support Lib. Asoc. 93, 1, 61-68.
the complex search strategies outlined above. [12] Henrich. A. and Morgenroth, K. 2007. Information retrieval
as e-learning course in German – Lessons learned after 5
The next stage in this work is to develop learning materials to
years of experienced. Proc. 1st international workshop on
deliver this learning framework, to engage in outreach activities
teaching and learning of information retrieval. Available on:
with users who undertake systematic reviews, and to provide them
http://tinyurl.com/z2ueueh.
with a structured learning framework that they can use to improve
their knowledge and skills. Guidance on how to develop learning [13] Sacchanand, C. and Jaroenpuntaruk, V. 2006. Development
objectives, curricula/learning materials, teaching methods and of a web-based self-traning package for information retrieval
assessment/feedback for each individual level of the search using the distance education approach. Elec. Lib. 24, 4, 501-
framework is provided in section 5. It is our plan to develop these 516.
concepts further. The proposed outcome of this work is to give [14] MacFarlane, A. 2011. Using multiple choice questions to
users the skills they need to be more effective searchers and to assist learning for Information Retrieval. In Efthimiadis, E.,
 Fernandez-Luna, J.M. Huete, J.F. and MacFarlane, A. Eds. [24] Glanville, J., Bayliss, S., Booth, A., Dundar, Y., Fleeman, N.,
Teaching and Learning in Information Retrieval. Springer Foster, L., Fraser, C., Fry-Smith, A, Golder, S., Lefebvre, C.,
Verlag, Berlin, 107-121. McNally, R., Miller, C., Paisley, S., Payne, L, Price, A.
[15] Zhu, L. and Tang, C. 2006. A module-based integration of Shaikh, H., Sutton, A., Welch, K. and Wilkinson, A. 2008.
information retrieval into undergraduate curricula. J. Comp. So many filters, so little time: The development of a Search
Sci. Col. 22, 2, 288-294. Filter Appraisal Checklist. J.MLA, 96,4, 356-361.

[16] Halttunen, E. and Sormunen, E. 2000. Learning information [25] Wilczynski, N.L., Haynes, R.B., Lavis, JN.,
retrieval through an educational game. Is gaming sufficient Ramkissoonsingh, R. and Arnold-Oatley, A. 2004. Optimal
for learning. Edu. Info, 18, 289-311. search strategies for detecting health services research
studies in MEDLINE. CMAJ 171, 10, 1179-1185.
[17] Taylor, R.S. 1968. Question-negotiation and information
seeking in libraries. College & Research Libraries, 29, 3, [26] Bates, J,. Bawden, D., Corderio, I. Steinerova, J., Vakkari, P.
178-194. and Vilar, P. 2005. Information Seeking and information
Retrieval. In Kajberg, L. and Lorring, L. Eds. European
[18] Belkin, N. J., Oddy, R.N. and Brooks, H.M. 1982. ASK for curriculum reflections on Library and information Science,
Information retrieval: part 1. background and theory. Denmark. RSLIS.
J.Documentation. 38, 2, 61-71.
[27] Wilson, T.D. 1999. Models in information behavior research.
[19] Dahlgren Memorial Library. 2016, Evidence-Based J.Doc, 55, 3, 249-270.
Medicine Resource Guide: Types of Clinical Questions.
Available on: http://tinyurl.com/zfsa3ob. [28] Cohen, L.B. 2001. 10 tips for teaching how to search the
web. American Libs, Nov, 44-46.
[20] Ranganathan, S.R. 2006. Colon Classification (6th Edition).
Ess Ess Publications. New Dehli. [29] MacFarlane, A. 2009. Teaching mathematics for search using
a tutorial style of delivery. Inf Retrieval, 12, 162-178.
[21] Markey, K. and Cochrane, P. 1981. Online training and
practice manual for ERIC database searchers (2nd Edition). [30] Bhavnani, S., Drabenstott, K., and Radev, D. 2001. Towards
ERIC Clearinghouse on Information Resources, Syracuse a united framework of IR tasks and strategies. In Proc.
University. Available on: http://tinyurl.com/j5v65wb. ASSIT annual meeting, 38, 340-354.

[22] ProQuest Dialog. Available on: http://tinyurl.com/l22vdk9. [31] Smith, G, Wood, L., Crawford, K., Coupland, M., Ball, G.
and Stephenson, B. 1996. Consructing mathematical
[23] Inskip, C. 2014. Information literacy is for life, not just a examinations to assess a range of knowledge and skills. Int J
good degree: a literature review. CILIP. Available on: Math Educ Sci Tech, 30, 47-63.
http://tinyurl.com/kjaujnh.

View publication stats