European Journal of Information Systems (2013) 22, 336–359

& 2013 Operational Research Society Ltd. All rights reserved 0960-085X/13
www.palgrave-journals.com/ejis/

A method for taxonomy development and

its application in information systems

Robert C. Nickerson1, Abstract

Upkar Varshney2 and A fundamental problem in many disciplines is the classification of objects in
a domain of interest into a taxonomy. Developing a taxonomy, however, is
Jan Muntermann3 a complex process that has not been adequately addressed in the information
1
systems (IS) literature. The purpose of this paper is to present a method for
Department of Information Systems, College taxonomy development that can be used in IS. First, this paper demonstrates
of Business, San Francisco State University,
through a comprehensive literature survey that taxonomy development in IS
San Francisco, California, USA; 2Department
of Computer Information Systems, Robinson
has largely been ad hoc. Then the paper defines the problem of taxonomy
College of Business, Georgia State University, development. Next, the paper presents a method for taxonomy development
Atlanta, Georgia; 3Department of Information that is based on taxonomy development literature in other disciplines and
Systems and Department of Finance, Accounting shows that the method has certain desirable qualities. Finally, the paper
and Taxes, University of Göttingen, Göttingen, demonstrates the efficacy of the method by developing a taxonomy in a
Germany domain in IS.
European Journal of Information Systems (2013) 22, 336–359.
Correspondence: R.C. Nickerson, doi:10.1057/ejis.2012.26; published online 19 June 2012
Department of Information Systems,
College of Business, San Francisco State
University, 1600 Holloway Avenue, Keywords: taxonomy; typology; classification; taxonomy development; research meth-
San Francisco, California 94132, U.S.A. odologies
Tel: þ 1-415-338-2138;
Fax: þ 1-415-405-0364;
E-mail: [email protected]
Introduction
A fundamental problem in many disciplines is the classification of objects
of interest into taxonomies. Biology has studied this problem extensively
and developed a number of classification schemes that order the
complexity of the living world and provide a foundation for biological
research. Similar schemes are also found in many social science fields.
Taxonomies play an important role in research and management because
the classification of objects helps researchers and practitioners understand
and analyze complex domains. This universal character of taxonomies is
also highlighted by Miller & Roth (1994, p. 286) who note that ‘taxonomies
y are useful in discussion, research and pedagogy’.
The role of taxonomies is also well recognized in the information systems
(IS) research literature. Glass & Vessey (1995) note that taxonomies provide
a structure and an organization to the knowledge of a field, thus enabling
researchers to study the relationships among concepts and, therefore, to
hypothesize about these relationships. McKnight & Chervany (2001) argue
that taxonomies can order otherwise disorderly concepts and allow
researchers to postulate on the relationships among the concepts. Williams
et al (2008) illustrate the use of taxonomies in understanding the science
behind design principles of observed artifacts. Fiedler et al (1996, pp. 11–12)
state that classification (i.e., taxonomy) has been important in research
Received: 24 November 2010 ‘since Aristotelian applications over 2000 years ago’. Sabherwal & King
Revised: 15 April 2012 (1995, p. 180) present a further argument by pointing out that ‘taxonomies
Accepted: 17 April 2012 also help us understand divergence in previous research findings’.
 A method for taxonomy development Robert C. Nickerson et al 337

From a philosophical foundations perspective, taxono- researchers in IS. This paper presents a survey of IS
mies are forms of conceptual knowledge in the epi- literature that identifies common themes related to taxo-
stemology of design science (Iivari, 2007), which also nomies and taxonomy development. On the basis of this
includes descriptive knowledge and prescriptive know- literature survey, we define the problem of taxonomy
ledge. As Iivari (2007, p. 46) explains ‘The research goal development. This problem definition serves as a guide
at the conceptual level is essentialist: concepts and for the creation of the taxonomy development method
conceptual frameworks at this level aim at identifying described in this paper.
essences in the research territory and their relationships’. After creating a taxonomy development method
Conceptual knowledge, including taxonomies, does not we need to demonstrate its efficacy by applying it to spe-
have a truth value but is relevant input for the deve- cific domains. This paper uses the method to develop
lopment of theories representing forms of descriptive a taxonomy in one IS domain, that of mobile applica-
knowledge, which have a truth value (Iivari, 2007). tions. We have chosen this domain because of its
Doty & Glick (1994) also argue that the classification of increasing importance and complexity with many new
objects (i.e., taxonomy) contribute to theory building. applications appearing regularly. Users, researchers, and
This point is also stressed by Bapna et al (2004, p. 23), developers need to be able to know where a new
who state that ‘a robust taxonomy can then be used to application fits with existing ones in this domain in
perform ex post theory building’. order to determine if it is something entirely new and
Taxonomy is a form of classification, and, as discussed unique, a significant variation of an existing application,
later, the terms, along with typology and framework, or just a retread of what we already have. A taxo-
are sometimes used interchangeably. Wand et al (1995, nomy provides a basis for making this determination
p. 291) note that classification is ‘a fundamental and could point out voids where new applications might
mechanism for organizing knowledge’. The systematic be developed.
organization of knowledge is a long running concern in Our research approach to creating a taxonomy deve-
IS (Hirschheim et al, 1995). Ontologies have been lopment method is based on the design science research
proposed as one way of dealing with this concern and paradigm, which aims to address new knowledge about
have found their way into IS (Guarino, 1998). Ontologies, artificial (i.e., manmade) objects that are designed to
defined by Gruber (1993) as explicit specifications of meet certain goals and provide utility to their users
conceptualizations, can include a number of artifacts (Simon, 1969). March & Smith (1995) present four kinds
besides formal ontologies including thesauri, controlled of contributions (artifacts) – constructs, models, methods,
vocabularies, folksonomies, and taxonomies (Gruninger and instantiations – and two processes (research activi-
et al, 2008). Often, however, these other artifacts – ties) – artifact building and artifact evaluation – that
including taxonomies – are viewed as distinct from characterize design science research in IS. In this paper,
ontologies (Dogac et al, 2002), although Wand & Weber we present a method that is intended to support design
(2004) note that theories of ontology sometimes function researchers during their research activities when deve-
like taxonomies. A taxonomy may be a step toward a loping a taxonomy for a specific domain. This method
future ontology, however, just as the periodic table, is an artifact that serves as a basis for future design
which can be viewed as a taxonomy of elements (Grove, science research, the purpose of which is to develop
2003), was a step toward various ontologies in chemistry new taxonomies. These new taxonomies are artifacts
(Pinto & Martins, 2004). Although we focus on taxonomy (models) in themselves. In terms of research processes, we
development in this paper, ontology development is first build an artifact (method) for developing taxo-
worthy of study in future research. nomies. Then we evaluate the artifact we have built
As we will show later in this paper, IS researchers have by using it to develop (i.e., build) a taxonomy that des-
proposed a number of taxonomies over the years. In many cribes and classifies existing or future objects in a specific
cases, however, the development of these taxonomies has domain. Since the result of this second step is an artifact
followed a largely ad hoc approach. Although the process (taxonomy), it is subject to evaluation. We evaluate the
of developing a taxonomy has been studied in a number taxonomy by assessing its efficacy in classifying objects of
of disciplines (e.g., Eldredge & Cracraft, 1980, and Sokal & interest in the specific domain.
Sneath, 1963, in biology; Bailey, 1994, in the social This paper is organized as follows. First, we define
sciences), little has been written about this process in the certain fundamental terms used in this paper. Then we
IS field. Glass & Vessey (1995) note the lack of a taxonomy discuss taxonomy development in other disciplines.
development methodology in their review of application- Next, we present our literature survey and our analysis
focused taxonomies. A well-conceived method for deve- of the papers surveyed. Then we present our problem
loping taxonomies would serve as a basis for developing statement for creating a taxonomy development method.
new taxonomies in IS that could bring order to complex Following these topics, we present our method for deve-
areas and potentially lead to new research directions. The loping taxonomies and justify it based on the foundation
general purpose of this paper is to present such a method. we have laid. Next, we demonstrate the use of our
Before creating a taxonomy development method, we taxonomy development method by developing a taxo-
need to examine how taxonomies are developed by other nomy of mobile applications. We conclude the paper

European Journal of Information Systems

338 A method for taxonomy development Robert C. Nickerson et al

with an extended discussion, summary of our results, and the system or process and sometimes used for the result
suggestions for future research. of applying the system (Bailey, 1994). We could refer to
the former as a taxonomic system and the later as
Classifications, frameworks, typologies, and a taxonomy, but we will follow the common practice of
taxonomies using the term taxonomy for both and allow the context
We are concerned with systems for grouping objects of to make it clear what we are referring to. In some
interest in a domain based on common characteristics. literature, taxonomy is restricted to empirically derived
We find that different terms are used for such systems, groupings, often found through cluster analysis or some
and that these terms are often confused. Before pro- other statistical technique. This form of taxonomy is
ceeding we need to clarify these terms and explain how sometimes called numerical taxonomy (Sokal & Sneath,
we use them in this paper. We note that this is not an 1963). Doty & Glick (1994) equate taxonomy with
easy task, or, as Sokal & Sneath (1963, p. 2) say ‘The ade- classification scheme, although they note that classifi-
quate definition of taxonomic terms would almost cation scheme, taxonomy, and typology are often used
require a book by itself’. interchangeably. Gregor (2006, p. 623) echoes this
The term classification is used to refer to both the system thought, stating that ‘the term typology is used more or
or process of organizing objects of interest and less synonymously for taxonomy and classifications’.
the organization of the objects according to a system. Bailey (1994) distinguishes taxonomies (classification
Bowker & Star (1999, p. 10) use the term classification for systems derived empirically) from typologies (classifi-
‘a spatial, temporal, or spatio-temporal segmentation cation systems derived conceptually). As we will see,
of the world’ and the term classification system for ‘a set however, Bailey presents a methodology for developing
of boxes (metaphorical or literal) into which things can taxonomies/typologies that is a combination of concep-
be put to then do some kind of work’. Bailey (1994, p. 1) tual and empirical approaches.
uses the term classification as the process of ‘ordering Much literature, however, uses taxonomy for systems
entities into groups or classes on the basis of similarity’. of groupings that are derived conceptually or empirically.
He says that classification can be unidimensional or We make this observation in the literature survey
multidimensional, and that it can be done conceptually discussed later in this paper. We also find in our literature
or empirically. Doty & Glick (1994), on the other hand, survey that taxonomy is by far the most common term
use the term classification scheme for a system that and thus we choose to use it throughout this paper
groups objects by applying specific ‘decision rules’. In this whether we are referring to a conceptually or empirically
paper, we use the term classification system for the derived grouping. We note that we could use any of
abstract groupings or categories into which we can put the terms discussed here – classification, framework,
objects and the term classification for the concrete result typology, or taxonomy – for the object of study in this
of putting objects into groupings or categories. paper, and we recognize that in some situations taxo-
Framework is another general term used for organizing nomy may not be the most precise term, but we opt for
objects. In their paper on framework and review articles, common recognition over precision in this paper and use
Schwarz et al (2007, p. 41) implicitly define a framework taxonomy exclusively.
in the context of framework articles as a ‘set of assump-
tions, concepts, values, and practices that constitutes Taxonomy development in other disciplines
a way of understanding the research within a body of Developing a taxonomy is a complex process. Biology,
knowledge’. Their definition is closest to that of a classi- with its well-known taxonomy of living organisms,
fication system discussed above and could be used provides some guidance. The traditional Linnaean taxo-
synonymously with it in some instances. They also pro- nomy, commonly found in biology textbooks, classifies
vide 10 purposes of a framework article and 17 qualities organisms based on a predefined hierarchy of categories
of a framework within their context, many of which over- from kingdom to species. Determining where a new
lap our formal definition of taxonomy and our necessary organism falls in the taxonomy involves identifying into
conditions for a taxonomy to be useful that we discuss which classification the organism fits at each level of the
later in this paper. Interestingly, they also provide what hierarchy. However, biological taxonomy development is
we would call a taxonomy for framework and review not limited to the traditional approach. Taxonomists also
articles with six dimensions. use phenetics and cladistics. Phenetics, sometimes called
The term typology is usually restricted to a system of numerical taxonomy, involves classifying organisms
conceptually derived groupings. Both Bailey (1994) and solely on the basis of their similarity. The researcher
Doty & Glick (1994) use the term this way. Bailey also identifies different characteristics of organisms and then
notes that typologies are usually multidimensional and uses statistical techniques to cluster the organisms into
distinguishes them from simple unidimensional classi- similar groups based on these characteristics (Sokal &
fication systems, implying that typologies are usually Sneath, 1963). In contrast, cladistics does not look at
more complex than classification systems. common characteristics but rather examines the evo-
The term taxonomy is perhaps the most confused. lutionary relationships among organisms (Eldredge &
As with classification, taxonomy is sometimes used for Cracraft, 1980). The researcher investigates the evolution

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 339

of organisms from others and then groups organisms (conceptual to empirical) to see how they fit with the
based on their evolutionary heritage. Two organisms may conceptualization. The other choice is to start with
be closely related in a cladistic taxonomy because they empirical data clusters and then to deductively con-
have a common ancestor even though they do not share ceptualize the nature of each cluster (empirical to
certain characteristics, thus putting them in different conceptual). We note that in this three-level model
groups in a phenetic analysis. Bailey combines typology development through con-
Taxonomy development in the social sciences has ceptualization with taxonomy development through
also been well studied. Bailey (1994) provides a thorough empirical data analysis to arrive at the final classification.
survey of the subject. As noted previously, Bailey distin-
guishes between a typology and a taxonomy, saying that
the former is derived conceptually or deductively and Survey of taxonomy development literature
the latter is derived empirically or inductively. In the In order to examine taxonomy development in IS, we
conceptual typology approach, the researcher proposes conducted a literature survey of papers that have a focus
a typology of categories or types based on a theoretical on the development of taxonomies. As a basis for the
ideal or model. In the process, the researcher could literature survey, we used the AIS Journal Rankings page
define an ideal type, which Bailey (citing Weber, 1949) available from the AIS website (http://ais.affiniscape
explains is the ‘extreme’ or ‘nirvana’ of types. The ideal .com/displaycommon.cfm?an ¼ 1&subarticlenbr ¼ 432).
type is used to examine empirical cases in terms of how From this ranking, we surveyed the top 30 journals and
much they deviate from the ideal. Alternatively, in the searched for papers that have the words taxonomy/ies
empirical approach the researcher proposes a taxonomy or typology/ies in their title and that were published
based on a constructed type, which, as Bailey (citing up to the year 2009. Further, we included papers
McKinney, 1966) explains, is not the ideal but based published in ICIS, AMCIS, ECIS, PACIS, and HICSS pro-
on reference to empirical cases. The constructed type is ceedings. We identified 73 relevant papers that propose
used to examine ‘exceptions’ to the type. Bailey compares new taxonomies.
the ideal type with the highest value in a set of data The AIS Journal Rankings focus on IS papers but
(assuming highest is best) and the constructed type to the also includes journals from computer science (CS) and
mean of the data (Bailey, 1994, p. 23). non-information systems business (Bus) disciplines.
In Bailey’s conceptual approach, the researcher deve- We included papers in these closely related research
lops a typology starting with a conceptual or theoretical fields to see how they compare. All papers surveyed
foundation and then derives the typological structure are listed in the Appendix of this paper. We classified
through deduction. The researcher may conceive of a each paper by its principal domain: IS, CS, and Bus. We
single type and then add dimensions until a satisfactorily recognize that the line between IS and CS is sometimes
complete typology is reached, a process called substruc- not clear. For borderline cases, we classified a paper as
tion (Bailey, 1994, p. 24). Alternatively, the researcher IS if it emphasized the organizational/managerial aspects
could conceptualize an extensive typology and then of the topic and as CS if it emphasized the technical
eliminate certain dimension in a process called reduction aspects. Papers in e-commerce (including mobile com-
(Bailey, 1994, p. 24) until a sufficiently parsimonious merce) were classified as IS. Bus papers include papers
typology is reached. in marketing, operations, management, and other areas
The conceptual approach is not based on empirical of business. The publishing journal also provided an
data, although such data could be brought in toward the indication of how to classify a paper. For example, ACM
end of the process for verification purposes. The empiri- journals generally publish papers in CS and journals such
cal approach, on the other hand, starts with data and as EJIS, MISQ, and ISR generally focus on IS research. We
derives the classification from this data using cluster identified 41 IS papers, 20 CS papers, and 12 papers in Bus
analysis or other statistical methods (Bailey, 1994, p. 34). fields.
The goal is to find similarities among the data and to For each paper we noted the type of taxonomy it
classify similar objects into the same category. Each cate- developed and the approach or method that the authors
gory in the resulting taxonomy is called a taxon (plural used for developing its taxonomy. We classified the
taxa). Using the concepts from biology, this approach is approach into one of the following categories:
phenetic.
Bailey (1984) describes the approaches just examined  Inductive
as different levels – conceptual and empirical – of a  Deductive
two-level model. Although researchers can approach  Intuitive
classification through either level, he suggests that a com- The inductive approach involves observing empirical
mon and often more useful approach is to use a three- cases, which are then analyzed to determine dimensions
level model that includes conceptual, empirical, and and characteristics in the taxonomy. The analysis may
indicator or operational levels. In this method the resea- be done using statistical techniques such as cluster
rcher has two choices. One is to start with the con- analysis or may use less rigorous techniques; we noted
ceptual approach and then to examine empirical cases this in our survey. This methodology is called phenetics

European Journal of Information Systems

340 A method for taxonomy development Robert C. Nickerson et al

or numerical taxonomy in biology. Bailey (1994) calls this Table 1 Taxonomy development in different domains
the empirical approach in sociology.
Taxonomy development approach
The deductive approach derives a taxonomy not from
empirical cases but instead from theory or conceptuali- Principal Inductive Inductive Deductive Intuitive Other
zation. It identifies dimensions and characteristics in the domain (statistical (informal (may be followed
taxonomy by a logical process derived from a sound analysis) analysis) by empirical
conceptual or theoretical foundation. Cladistics in bio- verification)
logy is similar to this approach. In sociology, Bailey IS 7 10 9 13 2
(1994) identifies this as the conceptual approach. This CS 1 3 6 10 0
approach may be followed by an analysis of empirical Bus 6 0 4 1 1
cases to evaluate and perhaps modify the taxonomy. Total 14 13 19 24 3
The intuitive approach is essentially ad hoc. The
researcher uses his or her understanding of the objects
to be classified to propose a taxonomy based on the approach. In total, we classified nearly one-third (24) of
researcher’s perceptions of what makes sense. There is no the surveyed papers as using an intuitive approach.
explicit method in this approach. Many papers do not base their taxonomy on a con-
Several other approaches were found that did not fall ceptual, theoretical, or empirical foundation. Although
into these categories including morphological analysis authors review the literature in their problem area, their
and the use of existing taxonomies. taxonomy is often not based on their literature review
Table 1 shows the distribution of approaches used in but instead is ad hoc. We classified these as using an
the IS, CS, and Bus papers that we surveyed. intuitive approach.
In a previous paper (Nickerson et al, 2010), we provide Of the papers that use an inductive approach (27),
a detailed analysis of 65 papers. For the current research about half (14) use statistical analysis to identify clusters
we excluded some papers that did not meet our selection appropriate for their taxonomy. The other half (13) use
criteria and identified additional papers that meet the informal techniques to examine their empirical cases.
criteria. The result is the 73 papers surveyed in this Papers that use a deductive approach (19) were hard to
research. The ease with which we found a large number of identify. Some of the papers that we identified as using an
papers that use the term taxonomy or typology in their intuitive approach may, in fact, use a deductive approach.
titles indicates to us that there is interest in classification We could not find any relationship between the deve-
schemes in IS and the other fields examined. lopment method used and the term – taxonomy or typo-
Of the papers we found, 56 use the term taxonomy logy – used for the final grouping. Although typologies
and 17 use the term typology. Overwhelmingly, the most are usually identified with a deductive approach and
common term used is taxonomy. However, there appears taxonomies with an inductive approach, this relationship
to be a great deal of confusion about what a taxonomy is. was not evident in the papers we surveyed.
Some papers seem to use the word taxonomy to show Papers in business tend to be more formal in their app-
that they are aware of the literature related to their pro- roach whereas papers in CS and IS tend to be less formal.
blem area. They classify the literature into two or three Papers in the IS domain use the most diverse taxonomy
simple categories, which may not completely define their development approaches.
domain. Other papers present lists as taxonomies, Few papers cite the taxonomy development literature
including lists of functions someone has to perform. from other disciplines that we have identified.
Published taxonomies range from very simple to A general conclusion from this survey is that many
complex. Some papers present simple N  N (N ¼ 2, 3, 4) researchers in IS find taxonomies useful, and that a
classifications. Most papers present taxonomies with four taxonomy development method that researchers can use
or fewer dimensions, but a few papers give taxonomies in place of ad hoc methods may be beneficial.
with more than 10 dimensions. There is no agreement on
what represents an appropriate number of dimensions.
Many papers provide little information about the Problem statement for taxonomy development
method the authors used to develop their taxonomies, In this section, we state the research problem that we are
so we could not identify the approach used in these exploring, that of defining a method for taxonomy
papers. In fact, we classified over 40% (30) of the development that can be used in the IS field.
surveyed papers as not identifying the method used. In To start we define what we mean by a taxonomy. A
some cases, we were able to infer the method from other taxonomy T is a set of it n dimensions Di (i ¼ 1, y, n) each
comments in the paper. When we could not, we inter- consisting of ki (kiX2) mutually exclusive and collectively
preted these papers as using a purely intuitive approach exhaustive characteristics Cij (j ¼ 1, y, ki) such that each
based on the author’s perception of what is a good object under consideration has one and only one Cij for
classification for its intended purpose. We recognize that each Di. Stated another way,
our interpretation may be incorrect in some instances.
Several other papers were classified as using an intuitive T ¼ fDi ; i ¼ 1; . . . ; njDi ¼ fCij ; j ¼ 1; . . . ; ki ; ki X2gg

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 341

The mutual exclusive restriction means that no object (Parsons & Wand, 2008, p. 843). While this condition
can have two different characteristics in a dimension. The may be desirable for some uses of a taxonomy, it is not
collectively exhaustive restriction means that each object universally required. Many useful taxonomies have been
must have one of the characteristics in a dimension. developed that do not meet this condition, including, we
Together these conditions mean that each object has contend, the Linnaean taxonomy of biology.
exactly one of the characteristics in each dimension. Without detailed guidance from the literature we are
We find a number of other terms used for dimension left on our own to define a useful taxonomy. We propose
and characteristic. Sometimes dimension is called vari- that a useful taxonomy has the following qualitative
able and the characteristics of a dimension are the possi- attributes:
ble values (domain) of the variable. This terminology
is common in the cluster analysis literature (e.g.,  It is concise: A useful taxonomy should be parsimonious,
Anderberg, 1973; Aldenderfer & Blashfield, 1984). Sokal for, as Bailey (1994) notes, lack of parsimony is a
& Sneath (1963), in their foundational book on numeri- weakness. A taxonomy should contain a limited number
cal taxonomy, use the terms taxonomic character and of dimensions and a limited number of characteristics in
character state, which are standard terms in biology. each dimension, because an extensive classification
Doty & Glick (1994) use attribute and value, respectively. scheme with many dimensions and many characteristics
Bailey (1994) uses dimension for typologies and vari- may exceed the cognitive load of the researcher and thus
able for taxonomies. In our survey of the literature be difficult to comprehend and apply. We could state
we found a variety of terms including category and this attribute formally as a function of the number of
capability, and characteristic and dimension with their dimensions and the number of characteristics that must
meanings reversed. We choose to use dimension and cha- have values less than maximums defined by factors
racteristic as above because they are at least as common including cognitive capacity in decision making. We
as others, they can apply to all forms of classification, and leave this analysis for future research.
they are descriptive.  It is robust: A useful taxonomy should contain enough
We want to develop useful taxonomies, but not nece- dimensions and characteristics to clearly differentiate
ssarily ‘best’ or ‘correct’ ones, as these cannot be defined the objects of interest. A taxonomy with few dimen-
and, in fact, may be moving targets that could change sions and characteristics may not be able to adequately
over time. In the design science literature, this problem of differentiate among objects. For example, a taxonomy
not being able to find an optimal solution is described as with only one dimension and two characteristics
design as a search process. As stated by Hevner et al (2004, within that dimension would not usually be useful.
p. 88), ‘The search for the best, or optimal, design is Bailey (1994, p. 1) makes this clear when he says that
often intractable for realistic information systems pro- the goal is to ‘make groups that are as distinct (non-
blems’. Instead, the search process attempts to discover overlapping) as possible, with all members within a
effective – or useful – solutions. The taxonomy develop- group being as alike as possible’. This attribute can
ment literature gives us little help with metrics for conflict with the conciseness attribute. As with the
evaluating taxonomies regarding effectiveness or useful- conciseness attribute, we could state the robustness
ness. Indeed, Bailey (1994, p. 2) makes this clear when he requirement as a function of the number of dimen-
repeatedly asks which of his example classifications is sions and the number of characteristics that must have
‘best’ without giving guidance for finding the answer values greater than minimums needed to characterize
other than saying that ‘a classification is no better the objects of interest. Again, we leave this analysis for
than the dimensions or variables on which it is based’. future research.
Later, he lists ‘weaknesses’ of typologies including lack  It is comprehensive: This attribute can be interpreted
of mutual exclusivity and collective exhaustivity; lack of two ways. One interpretation is that a useful taxonomy
parsimony; lack of changeability (i.e., they are static); can classify all known objects within the domain under
based on criteria that are arbitrary or ad hoc; and considerations. This corresponds to Bowker & Star’s
descriptive rather than explanatory (Bailey, 1994, p. 34). (1999) requirement of completeness. Taxonomies that
We note that we found these weaknesses in some of the are developed empirically should display this attribute.
proposed taxonomies surveyed previously. Bowker & Star The second interpretation is that a useful taxonomy
(1999) also note the importance of mutual exclusivity in includes all dimensions of objects of interest. Doty &
a classification system. In addition, they include com- Glick (1994, p. 294) imply this when, discussing ideal
pleteness, in the sense of covering all objects in a domain, types in typologies, they say that ‘typologies must
as an important property of a classification system. provide complete descriptions of each ideal type using
Parsons & Wand (2008) propose that the ability to draw the same set of dimensions’. Taxonomies that are
inferences is a critical condition for a useful classification, developed conceptually should display this attribute.
although not specifically a taxonomy. The authors define  It is extendible: A useful taxonomy should allow for
an inference as ‘the ability to infer some properties of an inclusion of additional dimensions and new character-
instance by virtue of identifying it as a member of a class, istics within a dimension when new types of objects
without having to directly observe these properties’ appear. A taxonomy that is not extendible may soon

European Journal of Information Systems

342 A method for taxonomy development Robert C. Nickerson et al

become obsolete. Put another way, it is dynamic, not a specific domain of interest. The method should have
static. Bailey (1994) points out that lack of change- the following qualities:
ability is a weakness.
 It takes into consideration alternative approaches to
 It is explanatory: A useful taxonomy contains dimen-
taxonomy development. Because several approaches to
sions and characteristics that do not describe every
taxonomy development are used, and no single app-
possible detail of the objects but, rather, provide useful
roach has been determined to be ‘best’, any method
explanations of the nature of the objects under study
must be flexible enough to allow for the selection of an
or of future objects to help us understand the objects.
approach or combination of approaches that is appro-
A taxonomy that simply describes objects may be of
priate for the domain being studied.
interest initially but will have little value in under-
 It reduces the possibility of including arbitrary or ad
standing the objects being classified. Bailey (1994)
hoc dimensions and characteristics in the taxonomy.
notes that typologies that are descriptive rather than
Any taxonomy should have dimensions and character-
explanatory are weak. This attribute allows a taxonomy istics based on conceptual and/or empirical grounds.
to be used to identify where an object is found in Arbitrary or ad hoc dimensions and characteristics
the taxonomy or to identify the characteristics of an should be avoided and a taxonomy development
object found in the taxonomy. That is, if someone method must support this goal.
knows the characteristics of an object, he/she will find  It can be completed in a reasonable period of time. Any
the object in an identifiable spot in the taxonomy, or if method must have a way of determining when it is
someone finds an object in a specific spot in the finished. There must be an ending condition in the
taxonomy, he/she will be able to identify the char- taxonomy development method that says when to
acteristics without having to know the complete stop, and this ending condition must be reachable in a
details of the object. reasonable amount of time.
 It must be straightforward to apply. Because taxo-
These attributes form the necessary conditions for a
nomies are developed by researchers with different
taxonomy to be useful, but they do not necessarily
levels of understanding of the taxonomy development
identify the sufficient conditions. They can, however,
literature, any method must be relatively easy to
give guidance to researchers and represent foundations
understand and apply without reference to the litera-
that can be used for descriptive evaluations on the basis
ture.
of informed argument by developing convincing argu-
 It must lead to a useful taxonomy. Since our goal is to
ments for a taxonomy’s utility (Hevner & Chatterjee,
develop useful taxonomies, any method must accom-
2010, p. 119). We are not able at this time to give
plish this goal.
sufficient conditions other than to say that a taxonomy is
useful if others use it. Clearly, this condition is tautolo- Our problem statement can thus be stated as follows:
gical. It is also correlated with design science research, Define a method for developing taxonomies such that
which seeks utility, not truth (Hevner et al, 2004). If this is
the only sufficient condition, however, then the only way  The resulting taxonomies satisfy the definition of a
to evaluate a taxonomy’s usefulness is to observe its use taxonomy given previously.
over time. We would like to have sufficient conditions  The resulting taxonomies have the qualitative attri-
that are easier to apply than this condition and butes listed previously.
that could be applied before putting a taxonomy into  The method has the qualities listed previously.
use. However, such sufficient conditions are likely to
depend on the expected use of a taxonomy. For example,
one use of a taxonomy might be to help users navigate Taxonomy development method
through a knowledge domain. A sufficient condition We now present our method for developing taxonomies
for this use might be related to how easy it is for the of objects in a domain of interest. Following the design
user to find related objects grouped together in the science paradigm, we are building an artifact that is a
taxonomy. Another use might be to discover new things method, the purpose of which is to build (develop)
about a domain. In this case, a sufficient condition another artifact (a taxonomy). Our method is intended to
might be that some observations can be made about the provide guidance for researchers during the design
domain that were not possible before. We could argue process of taxonomy development. We follow the defini-
that the necessary conditions given previously are also tion of March & Smith (1995, p. 257), who define a
sufficient but we feel that these conditions are not method as a ‘set of steps (an algorithm or guideline) used
adequate for sufficiency. We leave this as an area for to perform a task’. Our method should provide a guide-
future research. line to support the process of developing taxonomies in a
A taxonomy development method should have certain domain of interest, that is, it should provide ‘means to
qualities. The goal of such a method is to develop a taxo- reach desired ends while satisfying laws in the problem
nomy with a set of dimensions each consisting of a set environment’ (Hevner et al, 2004, p. 88). For an earlier
of characteristics that sufficiently describes the objects in version of our method see Nickerson et al (2009).

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 343

Meta-characteristic Although ideally the meta-characteristic should be

The development of a taxonomy involves determining specified before determining the characteristics in the
the characteristics of the objects of interest. The choice of taxonomy our experience has been that the meta-
the characteristics in a taxonomy is a central problem in characteristic sometimes does not become clear until
taxonomy development. The characteristics could be part way through the taxonomy development process
based on a theory but in reality any ‘theory’ is often when we ask ourselves what the overall ‘theme’ is of the
implicit (Aldenderfer & Blashfield, 1984). The researcher characteristics that we have proposed. We have found
must avoid, however, the situation of ‘naı̈ve empiricism’ that this exercise often leads to a clear statement of the
in which a large number of related and unrelated meta-characteristic and to eliminating some character-
characteristics are examined in the hope that a pattern istics and identifying new characteristics.
will emerge (Aldenderfer & Blashfield, 1984, p. 20). We see meta-characteristics appearing in research
To avoid this situation and provide a basis for identi- that develops taxonomies for various purposes, although
fying the characteristics of the taxonomy, we specify they are not identified as such. For example, Nickerson
a meta-characteristic at the beginning of the taxonomy (1997) develops a taxonomy of collaborative applications
development process. The meta-characteristic is the most based on the meta-characteristic of communication
comprehensive characteristic that will serve as the basis among group members. Williams et al (2008) choose
for the choice of characteristics in the taxonomy. Each two meta-characteristics – design and objectives – in
characteristic should be a logical consequence of the developing their taxonomy of digital services. Leem et al
meta-characteristic. (2004) develop a classification scheme for mobile busi-
The choice of the meta-characteristic should be based ness models starting with the meta-characteristic of
on the purpose of the taxonomy. For example, assume ‘business players’.
that the researcher is trying to classify computer plat-
forms (hardware and operating system) into a taxonomy. Ending conditions
If the researcher’s purpose is to distinguish platforms The method that we describe is iterative and thus must
based on processing power, then the meta-characteristic have conditions to determine when to terminate. These
is the hardware and software characteristics, such as CPU conditions are both objective and subjective. A funda-
power, memory, and operating system efficiency that mental objective ending condition is that the taxonomy
impact measures of power such as speed and capacity. On must satisfy our definition of a taxonomy, specifically
the other hand, if the researcher’s purpose is to dis- that it consists of dimensions each with mutually
tinguish among computer platforms based on how users exclusive and collectively exhaustive characteristics.
use them, then the meta-characteristic is the capability We have identified eight additional objective ending
of the platform to interact with users, such as the maxi- conditions listed in Table 2. Some of these conditions
mum number of simultaneously running applications are adapted from Sowa & Zachman’s (1992) rules for
and the user interface. their IS architecture framework. This list is not exhaustive
The purpose of the taxonomy should, in turn, be based and future research may identify additional objective
on the expected use of the taxonomy and thus could be ending conditions. An initial step for the researcher is to
defined by the eventual users of the taxonomy. The decide which of these or other objective conditions will
design process could involve first identifying the user(s) be used to determine when to terminate the method.
of the taxonomy who then specify the projected use of Subjective ending conditions also need to be exam-
the taxonomy, either explicitly or implicitly. Explicitly, ined. Previously, we noted that necessary conditions for a
the potential use of a taxonomy could be elicited from useful taxonomy are that it is concise, robust, compre-
actual users using elicitation techniques similar to those hensive, extendible, and explanatory. These conditions
employed in requirements analysis (see, e.g., Goguen & are the minimal subjective ones that must be met for the
Linde, 1993). Alternatively, the researcher could project method to terminate. Table 3 lists these subjective
who the users could be and decide, based on experience, conditions with questions that the researcher could ask
what use the users could make of the taxonomy. In the about each condition. The researcher can refer to the
computer platform example in the previous paragraph, previous discussion of these conditions for further
the researcher may wish to develop a taxonomy to be guidance. The researcher may wish to add more sub-
used by customers purchasing computers (the users of the jective conditions to these based on the researcher’s
taxonomy). If the researcher projects that these custo- particular view. The researcher needs to be able to argue
mers will be technology-savvy individuals interested in that all subjective conditions have been met before
processing power, then the first taxonomy would be terminating the method.
appropriate. On the other hand, if the researcher deter- Depending on the chosen ending conditions, the
mines that the customers will be application-savvy method may generate somewhat different taxonomies,
individuals interested in how they can use the computer, which is consistent with the design science philosophy of
then the second taxonomy would be appropriate. searching for useful, not necessarily optimal, solutions
The choice of the meta-characteristic must be done (Hevner et al, 2004). Our method can be extended to
carefully as it impacts critically the resulting taxonomy. select a more useful taxonomy among multiple choices

European Journal of Information Systems

344 A method for taxonomy development Robert C. Nickerson et al

Table 2 Objective ending conditions

Objective ending condition Comments

All objects or a representative sample of objects have been If all objects have not been examined, then the additional objects
examined need to be studied
No object was merged with a similar object or split into If objects were merged or split, then we need to examine the impact
multiple objects in the last iteration of these changes and determine if changes need to be made in the
dimensions or characteristics
At least one object is classified under every characteristics of If at least one object is not found under a characteristic, then the
every dimension taxonomy has a ‘null’ characteristic. We must either identify an
object with the characteristic or remove the characteristic from the
taxonomy
No new dimensions or characteristics were added in the last If new dimensions were found, then more characteristics of the
iteration dimensions may be identified. If new characteristics were found,
then more dimensions may be identified that include these
characteristics
No dimensions or characteristics were merged or split in the If dimensions or characteristics were merged or split, then we need
last iteration to examine the impact of these changes and determine if other
dimensions or characteristics need to be merged or split
Every dimension is unique and not repeated (i.e., there is no If dimensions are not unique, then there is redundancy/duplication
dimension duplication) among dimensions that needs to be eliminated
Every characteristic is unique within its dimension (i.e., there If characteristics within a dimension are not unique, then there is
is no characteristic duplication within a dimension) redundancy/duplication in characteristics that needs to be elimi-
nated. (This condition follows from mutual exclusivity of character-
istics.)
Each cell (combination of characteristics) is unique and is not If cells are not unique, then there is redundancy/duplication in cells
repeated (i.e., there is no cell duplication) that needs to be eliminated

Table 3 Subjective ending conditions

Subjective ending Questions
condition

Concise Does the number of dimensions allow the taxonomy to be meaningful without being unwieldy or
overwhelming? (A possible objective criteria for this condition is that the number of dimensions falls in the
range of seven plus or minus two; Miller, 1956.)
Robust Do the dimensions and characteristics provide for differentiation among objects sufficient to be of interest?
Given the characteristics of sample objects, what can we say about the objects?
Comprehensive Can all objects or a (random) sample of objects within the domain of interest be classified? Are all
dimensions of the objects of interest identified?
Extendible Can a new dimension or a new characteristic of an existing dimension be easily added?
Explanatory What do the dimensions and characteristic explain about an object?

and even merge multiple taxonomies into one if needed. (Bailey, 1994, pp. 31–32). Bailey’s approach is also static
We leave this for future research. in the sense that it terminates after applying one or the
other strategy and does not cycle back for additional
Taxonomy development method applications of the strategies. Thus, Bailey’s approach is
We are interested in the characteristics of the objects not consistent with the Hevner et al (2004, p. 88) design
being examined, not their evolutionary heritage. Thus, science research guideline that asks for ‘design as a search
our approach to developing a taxonomy is phenetic, process’.
not cladistic. We find that Bailey’s (1984) three-level Our method goes beyond Bailey’s concept to com-
indicator model provides a basis for a method for bine the conceptualization/deduction and empiricism/
developing taxonomies as it offers alternative app- induction strategies into a single method that encourages
roaches that involve both conceptualization/deduction the researcher to use the strategies in an iterative manner
and empiricism/induction. Bailey, however, implies that to best reach a useful taxonomy. In addition, our method
taxonomy development takes one approach or the includes specific ending conditions that test the taxo-
other – the ‘classical strategy’ of conceptual to empirical nomy as it is being developed. This approach is consistent
or the ‘opposite strategy’ of empirical to conceptual with the design science ‘generate/test cycle’ described by

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 345

Start

1. Determine meta-characteristic

2. Determine ending conditions

Empirical-to-conceptual Conceptual-to-empirical
3. Approach?

4e. Identify (new) 4c. Conceptualize (new)

subset of objects characteristics and dimensions of objects

5e. Identify common characteristics 5c. Examine objects for these

and group objects characteristics and dimensions

6e. Group characteristics into

dimensions to create (revise) 6c. Create (revise) taxonomy
taxonomy

7. Ending conditions met?

No
Yes
End

Figure 1 The taxonomy development method.

Hevner et al (2004, pp. 88–89). Finally, our method adds On the other hand, if the researcher has little under-
the important concept of meta-characteristic that Bailey standing of the domain but significant data about the
does not identify explicitly or implicitly. objects is available, then starting with the empirical-to-
Figure 1 shows the method that we propose. Steps in conceptual approach is appropriate. If the researcher has
this figure are numbered for later reference. A step-by- both significant knowledge of the domain and significant
step explanation follows the figure. data available about the objects, then the researcher will
The first step is to identify the meta-characteristic, have to use individual judgment to decide which app-
which, as discussed previously, is based on the purpose of roach is best. In the fourth case, where the researcher has
the taxonomy and in turn based on the users and their little knowledge of the domain and little data available,
expected use of the taxonomy. Next, the conditions that the researcher should investigate the domain of interest
end the process need to be determined. As discussed more before attempting to develop a taxonomy for it.
previously, there are both objective and subjective ending In subsequent iteration the researcher may choose to use
conditions. The researcher has a number of objective a different approach in order to view the taxonomy from
conditions that can be applied (Table 2). The subjective a different perspective and possibly gain new insight
ones are the most difficult to identify and to apply. about the taxonomy.
Table 3 provides initial guidance but the experience of In the empirical-to-conceptual approach, the research-
the researcher will have an impact on the selection of er identifies a subset of objects that he/she wishes
subjective conditions. In the case of multiple researchers to classify. These objects are likely to be the ones with
developing a taxonomy, various collaborative techni- which the researcher is most familiar or that are most
ques, including the Delphi method, could be used to easily accessible, possibly through a review of the lite-
determine these conditions. rature. The subset could be a random sample, a syste-
After these steps the researcher can begin with either an matic sample, a convenience sample, or some other type
empirical approach or a conceptual approach. The choice of sample. Next, the researcher identifies common cha-
of which approach to use depends on the availability racteristics of these objects. The characteristics must
of data about objects under study and the knowledge be logical consequences of the meta-characteristic.
of the researcher about the domain of interest. If little Thus, the researcher starts with the meta-characteristic
data are available but the researcher has significant and identifies characteristics of the objects that follow
understanding of the domain, then starting with the from the meta-characteristic. The characteristics must,
conceptual-to-empirical approach would be advised. however, discriminate among the objects; a characteristic

European Journal of Information Systems

346 A method for taxonomy development Robert C. Nickerson et al

that has the same value for all or nearly all objects is of be met so the process is repeated. In subsequent iterations
no use in the taxonomy even if it does follow from the the objective conditions must be evaluated and if not
meta-characteristic (Anderberg, 1973). The knowledge met, the process is repeated. If the objective conditions
and intuition of the researcher or other experts will be have been met, then the subjective conditions need to be
needed to identify the characteristics. If multiple resea- examined. Evaluating these conditions requires the
rchers or experts are working on the taxonomy, a group insight, experience, and skill of the researcher. Examples
methodology, such as the Delphi method, could be of heuristics that could be used were given previously. If
employed. In the process, characteristics may be pro- all the subjective conditions have not been met, then the
posed that turn out not to be relevant and thus can be process is repeated.
eliminated after further analysis. In repeating the method, the researcher must again
Once a set of characteristics has been identified, they decide which approach to use. Since new objects may
can be grouped formally using statistical techniques have been identified or new domain knowledge may
or informally using a manual or graphical process. The have been obtained in the previous iteration, the resea-
resulting groups form the initial dimensions of the rcher can use the previous heuristics anew to decide
taxonomy. This grouping involves creating ‘conceptual which approach to apply in the next iteration. In the
labels’ (Bailey, 1994, p. 32) for sets of related character- empirical-to-conceptual case the researcher examines
istics, that is, for the dimensions. Each dimension new objects to determine whether the existing character-
contains characteristics that are mutually exclusive and istics are sufficient to describe them or if new character-
collectively exhaustive. For example, dimension D1 may istics and possibly new dimensions are needed. As before,
group characteristics C11 and C12 and dimension D2 may statistical techniques can be used to aid in this process.
group characteristics C21, C22, and C23. All objects have This process could even result in the elimination of
one and only one of the characteristics C1j in dimension some dimensions and/or characteristics if they are deter-
D1 and one and only one of the characteristics C2j in mined not to be applicable. The result is the next version
dimension D2. Some dimensions may be dichotomous of the taxonomy. In the conceptual-to-empirical case the
(e.g., D1) and some may not be (e.g., D2). This process is researcher reviews the previous taxonomy to try to
based on the (limited) empirical data that has been identify additional conceptualizations that might not
gathered about objects and the deductive conceptuali- have been previously identified. In the process new
zation of the researcher. The result of this process is an characteristics may be deduced that fit into existing
initial taxonomy based on an empirical-to-conceptual dimensions or new dimensions may be conceptualized
approach. each with their own set of characteristics. It may even be
In the conceptual-to-empirical approach, the resea- the case that some dimensions or characteristics are
rcher begins by conceptualizing the dimensions of the eliminated or combined so that fewer dimensions and/or
taxonomy without examining actual objects. This process characteristics result. The researcher examines empirical
is based on the researcher’s notions about how objects cases using the new characteristics and dimensions to
are similar and how they are dissimilar. Since this is determine their usefulness in classifying objects. The
a deductive process, little guidance can be given other result is the next version of the taxonomy. As before, the
than to say that the researcher uses his/her knowledge of researcher checks if the ending conditions (objective and
existing foundations, experience, and judgment to subjective) have been met and either terminates the
deduce what he/she thinks will be relevant dimensions. process or repeats it.
Each dimension contains characteristics that must be We note that with each iteration of the design process,
logical consequences of the meta-characteristic. Thus, a new dimensions may be added and existing dimensions
test of the appropriateness of a dimension is whether may be eliminated. These are the processes of substruc-
its characteristics follow from the meta-characteristic. tion and reduction, respectively, described previously and
In the process, the researcher may propose dimensions discussed by Bailey (1994, p. 24).
that are not appropriate and thus can be eliminated. The It is important throughout the process that the resea-
researcher then examines objects for these dimensions rcher remembers that the taxonomy must be explana-
and characteristics. Are there objects that have each tory, not descriptive. That is, it must contain dimensions
of the characteristics in each dimension? If not, then the and characteristics that do not describe objects in
dimension may not be appropriate. As before, each complete detail but, rather, provide useful explanations
dimension must contain characteristics that are mutually of the nature of the objects under study or of future
exclusive and collectively exhaustive. The result of this objects.
process is an initial taxonomy based on a conceptual-to- Upon completion of the method (i.e., after the design
empirical approach. science building phase), the resulting taxonomy needs to
At the end of either of these steps, the researcher asks be evaluated for its usefulness (the design science
if the ending conditions have been met with the current evaluation phase). As we explained earlier, determining
version of the taxonomy. Both objective and subjective sufficient conditions for usefulness is difficult and
conditions must be checked. Since this is the first itera- evaluating usefulness may come down to seeing if others
tion, it is likely that none of the objective conditions will use it. Before this takes place, however, we can speculate

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 347

on potential use of the taxonomy. Given a user or users it to develop another artifact, specifically a model, and
and the purpose of the taxonomy, can the user’s purpose evaluating that artifact (taxonomy) by using it to classify
be satisfied with the taxonomy? To answer this question, objects of interest. For less detailed discussions of the
we could query users about their potential use of taxonomy developed here see Nickerson et al (2007) and
the taxonomy, or if users are not available, we could Nickerson et al (2009).
evaluate what the taxonomy tells the users in relation to We define a mobile application as a use of a mobile
the purpose of the taxonomy. technology by an end-user for a particular purpose, for
We do not propose that our approach is the best or only example, purchase a ring tone, check a weather fore-
taxonomy development method, only that it provides cast, transfer funds at a bank, make an airline reservation
guidance during the process of taxonomy development. and so on. Mobile applications are provided by mobile
Further, it follows from the taxonomy development services that have the infrastructure necessary to deliver
literature and satisfies the criteria listed previously in the application. A mobile service, however, may provide
our problem statement. Specifically, several different applications under the umbrella of one
service. For example, a mobile service may provide infor-
 It takes into consideration alternative approaches
mation about popular music and sell MP3 music files. For
to taxonomy development. Our method uses both
this paper, we view these as two different applications –
an empirical/inductive approach and a conceptual/
one, an informational application, and the other, a trans-
deductive approach and allows the researcher to decide
actional application – both provided by one service.
what approach to use at each pass through the
A number of taxonomies in the mobile/wireless area
method.
have been proposed. A review and critique of some
 It reduces the possibility of including arbitrary or ad
taxonomies presented in the early stages of the mobile
hoc dimensions and characteristics in the taxonomy.
era and predominantly written in German can be found
Our method requires that the characteristics and
in Lehmann & Lehner (2002). More recent papers include
dimensions be developed using a systematic process,
Dombroviak & Ramnath (2007), which gives a taxonomy
not developed in an ad hoc way.
of what the authors call ‘mobile pervasive’ applications,
 It can be completed in a reasonable period of time. Our
Leem et al (2004) and Abdelaal & Ali (2007), which
method is designed to reach closure in a few repeti-
presents taxonomies of mobile business models, Nysveen
tions of the method, although this requires the insight
et al (2005) and Heinonen & Pura (2006), which describe
of the researcher. If the researcher finds that the
taxonomies of mobile services, Williams et al (2008),
taxonomy is not converging on the ending con-
which gives a taxonomy of digital (not just mobile)
ditions in each repetition of the method, then the
services, Kemper & Wolf (2002), which presents a
researcher must take steps to rectify the situation,
taxonomy dealing with mobile application development,
possibly starting again from scratch.
and Dobson (2004), which gives a taxonomy of location
 It is straightforward to apply. Our method provides a
in pervasive computing.
specific set of steps that a trained researcher should be
The users of the taxonomy we develop are researchers
able to apply without additional reference to the
and developers of mobile applications. In characterizing
taxonomy development literature.
mobile applications, this user group is interested in high-
 It leads to a useful taxonomy. This criteria is the
level characteristics of the user interaction with mobile
hardest to determine if it has been satisfied. As
applications. They are not interested in technical char-
discussed previously, a useful taxonomy must meet
acteristics of the application, such as type of mobile
certain necessary conditions. The subjective ending
device used or speed of network connection, nor in how
conditions of the method include these conditions.
the user uses technology, such as keypads and touch
These conditions, however, are not necessarily suffi-
sensitive screens, with the application. Indeed, mobile
cient for a useful taxonomy. Unless more specific
technology is constantly evolving and any taxonomy
sufficient conditions can be identified, the only way
based on it may quickly be out of date. In addition, this
to determine if the resulting taxonomy is useful is to
user group is not interested in characterizing the pur-
observe its use by others over time. Asking users to
poses of mobile applications, although developing
evaluate the usefulness of a taxonomy is one way of
a taxonomy with this goal may be beneficial. This user
evaluating the taxonomy. If a taxonomy turns out not
group wants to be able to use the taxonomy to identify
to be useful, then the process needs to be restarted,
the characteristics of how users interact with applications
perhaps with a different meta-characteristic.
currently or may interact with applications in the future
at a higher level of abstraction than the physical inter-
Development of a taxonomy of mobile action with the application. Specifically, the purpose
applications of our taxonomy is to distinguish among mobile
To demonstrate the efficacy of the method described applications based on how the application user interacts
previously, we use it to develop a taxonomy of mobile at a high level with the application. Such a taxonomy will
applications. In the design science paradigm we are help researchers and developers identify whether new
evaluating the artifact (method) built previously by using applications are truly unique from the user’s perspective

European Journal of Information Systems

348 A method for taxonomy development Robert C. Nickerson et al

and where applications do not exist in the taxonomy Table 4 Taxonomy of mobile applications after
suggesting opportunities for new applications. Thus, the Iteration 1
meta-characteristic for the taxonomy development
Applications Temporal Communication
process is the high-level interaction between the user
and the application. S AS INF INT
We now demonstrate the application of the method
Mobile voice communications X X
shown in Figure 1 using the numbered steps in the figure.
Mobile messaging X X
Mobile TV X X
Step 1: Meta-characteristic: High-level interaction bet-
ween the application user and the application.
Step 2: Ending conditions: The method will end when meta-characteristic in the sense that they are
both objective and subjective conditions have aspects of the high-level user interaction with
been met. For simplicity in this example, we the application.
will use only two objective ending conditions
from Table 2, specifically, that no new dimen- Step 6e: Because the number of characteristics is small,
sions are added in the last iteration and no we can group these characteristics manually
additional applications need to be examined. into the following dimensions to form our first
Subjectively, the method will end when all the taxonomy:
conditions in Table 3 are met, that is, when the
taxonomy is determined to be concise, robust,  Temporal dimension: synchronous user in-
comprehensive, extendible, and explanatory. teraction and asynchronous user interaction
characteristics.
 Communication dimension: informational
Iteration 1: (information flows only from application to
user) and interactive (information flows both
Step 3: Approach: We decide to use the empirical-to- from application to user and from user to
conceptual approach first because we have application) characteristics.
identified some mobile applications from pre-
In the notation used previously for our definition of
vious research in the mobile area.
taxonomy, our first taxonomy T1 consists of dimension
Step 4e: We select the following convenience sample of
D1 ¼ Temporal with characteristics C11 ¼ Synchronous
mobile applications from the literature (Varsh-
and C12 ¼ Asynchronous, and D2 ¼ Communication with
ney & Vetter, 2002; Ngai & Gunasekaran,
characteristics C21 ¼ Informational and C22 ¼ Interactive,
2007):
or more simply:
 Mobile voice communications. T1 ¼ fTemporal ðSynchronous; AsynchronousÞ;
 Mobile messaging.
Communication ðInformational; InteractiveÞg
 Mobile TV.

Step 5e: We identify the following user interaction Table 4 shows the classification of the applications
characteristics in these applications based examined in this iteration in this taxonomy.
on our understanding of the applications
and identify which application has each char- Step 7: Ending conditions: Since two dimensions were
acteristic: created in this iteration, the method must be
repeated. In addition, more mobile applica-
 User interacts with application synchro- tions exist that need to be examined. We note,
nously. however, that the taxonomy is concise, exten-
 User interacts with application asynchro- dible, and explanatory, but its limited number
nously. of dimensions and characteristics may not be
 Information flows from the application to robust, and it is not known if it is comprehen-
the user. sive because more mobile applications exist
 Information flows from the user to the that need to be considered. At least one more
application and also from the application iteration is needed.
to the user.

For example, the mobile TV application in- Iteration 2:

volves synchronous user interaction and in-
formation flowing from the application to the Step 3: Approach: We decide to use the empirical-to-
user. All these characteristics follow from the conceptual approach again because we have

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 349

identified additional mobile applications from The other two characteristics can be grouped
previous research in the mobile area. into the following dimension:
Step 4e: We select the following sample of addi-
tional mobile applications from the literature  Transaction dimension: transactional (user
(Varshney & Vetter, 2002 and Ngai & Gunase- engages in financial transaction) and non-
karan, 2007): transactional (user does not engage in finan-
cial transaction) characteristics.
 Purchasing location-based contents.
At this point we have our second taxonomy:
 Mobile inventory management.
 Product location and tracking.
 Mobile advertising. T2 ¼ fTemporal ðSynchronous; AsynchronousÞ;
 Mobile navigation. Communication ðInformational; Reporting; InteractiveÞ;
Transaction (Transactional, Non-transactional)g
Step 5e: We identify the following user interaction cha-
racteristics in these applications based on our Table 5 shows the applications examined so far
understanding of the applications and identify classified in this taxonomy.
which application has each characteristic:
Step 7: Ending conditions: Since one more dimension
 Information flows from the application to
was created in this iteration, the method must
the user.
be repeated. In addition, more mobile applica-
 User engages in a financial transaction
tions exist that need to be examined. We note,
through the application.
however, that the taxonomy is concise, ex-
 User does not engage in a financial transac-
tendible, and explanatory, but its limited
tion through the application.
number of dimensions and characteristics
may not be robust and it is not known if it is
For example, the purchasing location-based
comprehensive because more mobile applica-
contents application involves synchronous
tions exist that need to be considered. At least
user interaction, information flowing from the
one more iteration is needed.
application to the user, and the user engaging in
a financial transaction. All these characteristics
follow logically from the meta-characteristic. Iteration 3:

Step 6e: We recognize that the first characteristic is an Step 3: Approach: For the third iteration we decide
additional characteristic in the communica- to use the conceptual-to-empirical approach in
tion dimension identified previously. Thus, order to get a different perspective on the
this dimension becomes: taxonomy.
Step 4c: We conceive that some applications can interact
 Communication dimension: informational with anyone, that is, they are public, and some
(information flows only from application to applications can only be used by individuals
user), reporting (information flows only from who have certain privileges such as those who
user to application), and interactive (informa- work for a company, that is, they are private. We
tion flows both from application to user and identify this as an access dimension and note
from user to application) characteristics. that it follows from the meta-characteristic:

Table 5 Taxonomy of mobile applications after Iteration 2

Applications Temporal Communication Transaction

S AS INF RP INT T NT

Mobile voice communications X X X

Mobile messaging X X X
Mobile TV X X X
Purchasing location-based contents X X X
Mobile inventory management X X X
Product location and tracking X X X
Mobile advertisement X X X
Mobile navigation X X X

European Journal of Information Systems

350 A method for taxonomy development Robert C. Nickerson et al

 Access dimension: public (can be used by Step 5e: We identify the following user interaction
anyone) and private (use restricted to certain characteristics in these applications and iden-
individuals) characteristics tify which application has each characteristic:

 Application has a single user.

Step 5c: We identify instances of these types of applica-
 Application has multiple users.
tion. For example, purchasing location-based
contents is a public application and mobile Although mobile applications can be used by
inventory management is a private applica- many users simultaneously, users may not be
tion. aware of this characteristic and view their use of
Step 6c: Adding this dimension to the previous three the application as individual. With some appli-
dimensions creates our next taxonomy: cations, however, users may know that they are
part of a multiple-user community using the
T3 ¼ fTemporal ðSynchronous; AsynchronousÞ; application.
Communication ðInformational; Reporting; InteractiveÞ;
Transaction ðTransactional; Non-transactional),
Step 6e: We can group these characteristics manually
Access ðPublic; PrivateÞg
into the following dimension to form our next
Table 6 shows the applications identified previously taxonomy:
classified with this taxonomy.
 Multiplicity dimension: individual (user ex-
periences the application as if he/she were
Step 7: Ending conditions: Since one dimension was
the sole user) and group (user views use of
added in this iteration, we must repeat the
the application as part of a group) character-
method. The taxonomy is concise, extendible,
istics
and explanatory. However, the addition of
another dimension makes the taxonomy At this point we have our next taxonomy:
more robust. At least one more iteration is
needed. T4 ¼ fTemporal ðSynchronous; AsynchronousÞ;
Communication ðInformational; Reporting; InteractiveÞ;
Transaction ðTransactional; Non-transactional),
Iteration 4: Access ðPublic; PrivateÞ;
Multiplicity ðIndividual; GroupÞg
Step 3: Approach: Since there are more applications
to examine, we follow the empirical-to- Table 7 shows all the applications identified so far
conceptual approach for this iteration. classified in this taxonomy.
Step 4e: We select the following additional mobile
applications from the literature (Varshney &
Vetter, 2002; Ngai & Gunasekaran, 2007): Step 7: Ending conditions: Since one more dimension
was created in this iteration, the method must
 Mobile games. be repeated. We note, however, that the
 Mobile entertainment services. taxonomy is concise, extendible, and explana-
 Mobile social networking. tory, and more robust than before. At least one
 Mobile communities. more iteration is needed.

Table 6 Taxonomy of mobile applications after Iteration 3

Applications Temporal Communication Transaction Access

S AS INF RP INT T NT PU PR

Mobile voice communications X X X X

Mobile messaging X X X X
Mobile TV X X X X
Purchasing location-based contents X X X X
Mobile inventory management X X X X
Product location and tracking X X X X
Mobile advertisement X X X X
Mobile navigation X X X X

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 351

Table 7 Taxonomy of mobile applications after Iteration 4

Applications Temporal Communication Transaction Access Multiplicity

S AS INF RP INT T NT PU PR I G

Mobile voice communications X X X X X

Mobile messaging X X X X X
Mobile TV X X X X X
Purchasing location-based contents X X X X X
Mobile inventory management X X X X X
Product location and tracking X X X X X
Mobile advertisement X X X X X
Mobile navigation X X X X X
Mobile games X X X X X
Mobile entertainment services X X X X X
Mobile social networking X X X X X
Mobile communities X X X X X

Iteration 5: content is location-based but mobile games are

not, and mobile social networking is identity-
Step 3: Approach: We decide to use the conceptual to based but mobile entertainment services are not.
empirical because we feel that we can conceive Step 6c: Adding these two dimensions to the previous
new dimensions. five dimensions gives us our next taxonomy:
Step 4c: We conceive of two more dimensions of
T5 ¼ fTemporal ðSynchronous; AsynchronousÞ;
mobile applications with characteristics related
to the meta-characteristic. Communication ðInformational; Reporting; InteractiveÞ;
Transaction ðTransactional; Non-transactional),
Some mobile applications may provide custo-
Access ðPublic; PrivateÞ;
mized information or functionality based on
the user’s location, whereas other applications Multiplicity ðIndividual; GroupÞ;
may not depend on where the user is located. Location (Location-based, Non-location-based),
The location dimension deals with whether the Identity (Identity-based, Non-identity-based)g
location of the user is used to modify the
interaction of the application with the user: All the applications are classified in this taxonomy in
Table 8.
 Location dimension: Location-based (applica-
tion uses the user’s location) and non-loca- Step 7: Ending conditions: Since we added two dimen-
tion-based (application does not use the sions in this iteration, we need to repeat the
user’s location) characteristics method. In addition, other applications need to
be examined. The current taxonomy is concise,
Like the location dimension, some mobile extendible, and explanatory, and the addition of
applications may adjust their information or two dimensions to a total of seven dimensions
functionality based on an awareness of who makes the taxonomy robust. It is not known if it
the user is, whereas other applications may is comprehensive because more mobile applica-
not depend on the user’s identity. The tions exist that need to be considered.
identity dimension relates to whether the
identity of the user is used to modify the way Iteration 6:
the application interacts with the user based
on the user’s identity: Step 3: Approach: Since there are more applications to
examine, we follow the empirical-to-concep-
 Identity dimension: Identity-based (applica- tual approach for this iteration.
tion uses the user’s identity) and non-iden- Step 4e: We identify additional applications from the
tity-based (application does not use the user’s literature to consider (Varshney & Vetter, 2002;
identity) characteristics. Ngai & Gunasekaran, 2007):

Step 5c: We find a number of applications from our  Mobile auctions and financial services.
original lists with these characteristics. For  Mobile distance education.
example, mobile purchasing of location-based  Mobile ticketing.

European Journal of Information Systems

352 A method for taxonomy development Robert C. Nickerson et al

Table 8 Taxonomy of mobile applications after Iteration 5

Applications Temporal Communication Transaction Access Multiplicity Location Identity

S AS INF RP INT T NT PU PR I G LB NLB I NI

Mobile voice communications X X X X X X X

Mobile messaging X X X X X X X
Mobile TV X X X X X X X
Purchasing location-based contents X X X X X X X
Mobile inventory management X X X X X X X
Product location and tracking X X X X X X X
Mobile advertisement X X X X X X X
Mobile navigation X X X X X X X
Mobile games X X X X X X X
Mobile entertainment services X X X X X X X
Mobile social networking X X X X X X X
Mobile communities X X X X X X X

Table 9 Taxonomy of mobile applications after Iteration 6

Applications Temporal Communication Transaction Access Multiplicity Location Identity

S AS INF RP INT T NT PU PR I G LB NLB I NI

Mobile voice communications X X X X X X X

Mobile messaging X X X X X X X
Mobile TV X X X X X X X
Purchasing location-based contents X X X X X X X
Mobile inventory management X X X X X X X
Product location and tracking X X X X X X X
Mobile advertisement X X X X X X X
Mobile navigation X X X X X X X
Mobile games X X X X X X X
Mobile entertainment services X X X X X X X
Mobile social networking X X X X X X X
Mobile communities X X X X X X X
Mobile auctions and financial services X X X X X X X
Mobile distance education X X X X X X X
Mobile ticketing X X X X X X X

Steps 5e and 6e: We cannot identify any new character- Our final taxonomy of mobile applications is given in
istics and dimensions from these appli- the previous formula (T5) and listed here:
cations. We group the new applications,
along with the previous applications,  Temporal dimension: Synchronous and asynchronous
using the existing characteristics and characteristics.
dimensions as shown in Table 9.  Communication dimension: Informational, reporting,
Step 7: Ending conditions: We have added no and interactive characteristics.
new dimensions with this iteration and  Transaction dimension: Transactional and non-trans-
we have examined a large sample of actional characteristics.
mobile applications. Hence, the objec-  Access dimension: Public and private characteristics.
tive ending conditions are met. The  Multiplicity dimension: Individual and group charac-
taxonomy is concise, extendible, robust, teristics.
and explanatory. With the considera-  Location dimension: Location-based and non-location-
tion of the additional applications, the based characteristics.
taxonomy appears to be comprehen-  Identity dimension: Identity-based and non-identity-
sive. Thus, the taxonomy meets the based characteristics.
subjective ending conditions. The As noted previously our goal is to create useful taxo-
method ends at this point. nomies. Our final test, then, is to examine the resulting

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 353

taxonomy for its usefulness for the intended users and about their lives to a group of geriatric friends who
the intended purpose. The users of the mobile applica- they may not be able to meet face to face.
tions taxonomy were projected to be researchers and
developers of mobile applications, and their purpose was Discussion
to distinguish among mobile applications based on how This paper has presented a method for taxonomy deve-
the application user interacts with the applications at lopment that is based on the taxonomy development
a high level so as to help the taxonomy users identify literature in other disciplines. The method is a hybrid of
the uniqueness of newly developed mobile applications methods used for typology development (conceptual)
and opportunities for new mobile applications. While the and methods used for taxonomy development (empiri-
former goal cannot be tested until new applications cal). The artifact resulting from applying our method can
appear, insight into the later goal can be gained by be thought of as a hybrid of a typology and a taxonomy.
examining the taxonomy in Table 9. We can make several It could be called a classification, framework, typology,
observations, including the following: taxonomy, or some other term, although we have chosen
to call it a taxonomy because our literature survey indi-
1. An approximately equal number of synchronous and cated that this term is the most commonly used one in
asynchronous applications are identified in Table 9, papers that develop this type of artifact. By presenting
implying that both modes have value. In the future, a hybrid approach resulting in hybrid taxonomies, we are
new applications could be developed that run in providing a method that results in taxonomies that are
synchronous mode, but if network infrastructure is likely to be more broadly useful than those that come
experiencing high traffic load, these applications from more restricted approaches.
could adjust to run in asynchronous mode. Our method does not identify an ideal type as is the
2. Only one application in Table 9 is reporting, which expected result in traditional typology development.
may be because the current needs of users for reporting Likewise, our method does not result in a purely con-
are being met by fixed devices. In the future, however, structed type as in taxonomy development. Rather, our
such requirements may move to mobile devices. Thus, method takes a pragmatic approach to create an artifact
more research and development could be done in that combines elements of both ideal and constructed
designing mobile applications that have the reporting types. We do not look at the effectiveness of individual
characteristic. elements classified in the taxonomy, as proposed by
3. Most applications in Table 9 are non-transactional, Doty et al (1993) in their discussion of ideal types and
which may be because financial transactions, such as organizational configurations, but rather at the overall
online payments, are difficult with mobile devices. effectiveness of a resulting taxonomy to classify objects in
There may be opportunities for research and develop- a domain.
ment into technology that facilitates mobile transac- The flexibility of our method allows the researcher to
tions, such as mobile payment systems. develop taxonomies without the limitations imposed by
4. Only two private applications are in Table 9, which traditional typology or taxonomy development. The
could be because most mobile applications today are artifacts resulting from our method are likely to be more
B2C with public access. There may be opportunities comprehensive and more extendable (important char-
for research and development in mobile B2B or B2E acteristics of a useful taxonomy) than those resulting
applications, which would have private access. from traditional methods. Traditional typologies, with
5. Table 9 includes only four group applications, which their emphasis on ideal types, may be less comprehensive
means that there may be opportunities to develop new and be harder to extend due to the difficulty in identi-
group applications. fying ideal types. Traditional taxonomies, with their
6. Although there are fewer location-based applications emphasis on constructed types, may also be less compre-
than non-location-based applications in Table 9, the hensive and be difficult to extend due to their reliance on
difference is small. Thus, both types of applications empirical cases. By combining the two approaches we
have value and future applications may be designed allow the researcher to use a mixture that best serves the
with either approach. researcher’s needs.
7. Identity-based and non-identity based applications Our method is derived from the taxonomy develop-
are almost equally balanced in Table 9, implying that ment literature in other disciplines, most notably the
both types of applications may be developed in the social sciences. It follows from Bailey’s (1984) ‘three-level
future. model’ but goes significantly beyond that model by
8. A number of voids can be found in the taxonomy. including alternative methods (conceptual to empirical
For example, there are no applications that have the and empirical to conceptual) that can be repeated in
combined characteristics of reporting, non-transac- different combinations. The iterative nature of our
tional, private, and group. An opportunity may exist method allows it to add/split and remove/merge dimen-
for applications to fill this and other voids. For exam- sions and characteristics as it converges on an artifact
ple, an application for older adults might be developed that is at the same time concise and robust. Our method
that allows users to report wellness and other information also includes the important concept of a meta-characteristic

European Journal of Information Systems

354 A method for taxonomy development Robert C. Nickerson et al

and objective and subjective ending conditions, elements this paper, we have used it to develop other taxonomies
that are not found in Bailey’s approach. Thus our and to critique published taxonomies and typologies.
method, while related to Bailey’s, is different and a We have also taught our method to other researchers
significant contribution to the research in taxonomy who have used it to develop taxonomies for their
development. research (Geiger et al, 2011; Krug et al, 2012). Continued
Even with the method that we propose the judgment efforts in these endeavors appear to be promising.
of the researcher is required. The selection of the
meta-characteristic, the determination and application Summary and conclusion
of the ending conditions, the decision of which path This paper has examined the question of taxonomy
(empirical to conceptual or conceptual to empirical) to development from several angles. First, it looked at a
take at each iteration, the identification of object range of literature and concluded that, whereas many IS
subsets, the conceptualization of characteristics and researchers have found taxonomies are useful, often the
dimensions, and other steps in the method all require process of developing a taxonomy in IS is ad hoc (unlike
human judgment. Indeed, in some cases conflicting taxonomy development in business or management-
criteria may have to be resolved by the researcher, such related outlets, which tend to use more formal app-
as potential conflicts in the necessary criteria for a roaches), and thus a method for taxonomy development
taxonomy to be useful. We have provided guidelines and that researchers can use in place of an ad hoc app-
heuristics to help the researcher, but these do not roach may be beneficial. Second, the paper defined the
supplant the researcher’s expertise and judgment. Some problem of taxonomy development, presenting necessary
tools may help, such as statistical cluster analysis for the conditions for a taxonomy to be useful and requisite
examination of empirical cases, but the researcher qualities of a taxonomy development method. Third, the
must make the final determination of the taxonomy’s paper presented a method for developing taxonomies
structure. based on well-established literature in taxonomy devel-
Researchers can use our method to develop taxonomies opment and showed that the method had the requisite
in different domains. Because the method is founded qualities. Fourth, the paper demonstrated the efficacy of
on established concepts about taxonomy development the method by developing a taxonomy in an IS domain.
and has certain desirable qualities, researchers will have The approach that the paper took followed the design
a high degree of confidence that taxonomies developed science paradigm by first building a method for taxo-
using this method will be useful to them and to others. nomy development, then evaluating the method by
We have demonstrated the use of the method in only using it to build a taxonomy (artifact), and finally
one domain, but we have used it in other domains, and evaluating the taxonomy by using it to classify objects
we are confident that the method can be applied in a in the domain.
wide range of domains. The most important contribution of this paper is
We have proposed our method for use in developing the method that we present for developing taxonomies.
taxonomies in IS and have illustrated it with an example With this contribution we address the dichotomy of
from this discipline because it is the discipline with design science research, since our method supports
which we are most familiar. There is nothing unique in design science researchers during their research activities
our method, however, to IS. Indeed, our problem state- (design as a process) in order to develop useful taxo-
ment for taxonomy development is not specific to IS, and nomies (design as an artifact). Our method contributes to
no steps in our method apply only to IS. Investigation the knowledge base of IS research, that is, the scientific
of the use of our method in other areas is a potentially foundations from which it can be drawn when develop-
fruitful area for future research. We speculate that this ing new taxonomies. The method addresses the pro-
research is likely to indicate the general applicability of cess of taxonomy development and provides guidance
our method, and, if so, this may be the most significant during the design science build/evaluate cycle of deve-
contribution of this paper. loping taxonomies and evaluating them against a set of
The implications of our method for researchers is that necessary conditions for usefulness.
it provides an approach to taxonomy development that Future research in taxonomies and taxonomy develop-
is neither intuitive nor ad hoc, as we found was the case ment in IS can take a number of directions. One is to
in many papers that we surveyed, but rather deliberate investigate the question of sufficient conditions for
and planned. Researchers can be reasonably confident a useful taxonomy, a question that was identified pre-
that the taxonomies developed using our method will viously in this paper. Along with this question goes the
meet their needs if not exactly then at least closely. question of whether a useful taxonomy has a minimal
Readers of papers that present taxonomies developed number of dimensions. Fundamental to the method
using our method can also be reasonably confident that we present in this paper is the concept of a meta-
the taxonomy presented was developed in an estab- characteristic. How to determine the appropriate meta-
lished way. characteristic for a taxonomy needs further investigation.
Although we have presented only one example of Determining and applying ending conditions in our
the use of our method in taxonomy development in taxonomy development method requires some subjective

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 355

evaluation. Investigating ways that these conditions can looking at the same domain. Then the question of how
be made more objective is worthy of further investigation. to merge the taxonomies becomes important. This avenue
As we have noted, there is no one best taxonomy. We find of research could also lead to investigation of group
further arguments for this in the design science founda- taxonomy development and whether a software tool,
tions, which underline that the search for an optimal perhaps employing expert collaboration using the Delphi
design is intractable. In fact, multiple taxonomies may be or some other method, might be useful. As we have
developed for a domain even when starting with the same pointed out, taxonomies are not static but change over
meta-characteristic. How to compare different taxonomies time as new objects that may or may not fit into an
for a given domain to determine which, if any, is best is existing taxonomy are developed or identified. Addressing
an open question. The method that we present results in this increased diversity of objects in taxonomy modifica-
a single taxonomy. An alternative method would be to tion is another area for future research. Finally, applying
develop several, possibly overlapping taxonomies for the method in this paper to various domains and
different subsets of a domain. These taxonomies could investigating the resulting taxonomies will be an ongoing
come from a single researcher or from different researchers area for research.

About the authors

Robert C. Nickerson is a Professor of Information technologies, pervasive computing, and m-commerce,

Systems at San Francisco State University and Chair and he has authored numerous papers. He chaired the
of the Department of Information Systems from 2006 to International Pervasive Health Conference in 2006 and
2012. His current research interests include taxonomies program chaired AMCIS in 2009.
and taxonomy development in information systems,
wireless/mobile systems, electronic commerce systems, Jan Muntermann is a Professor and Chair of Electronic
and crowdsourcing. He has been a regularly invited Finance and Digital Markets at the Faculty of Economic
professor at several European universities. Sciences, University of Göttingen. His research interests
include decision support systems, design science and IT
Upkar Varshney is an Associate Professor of CIS at Governance, especially in the fields of E-Finance and
Georgia State University, Atlanta. His current interests Electronic Markets. His research has appeared in outlets
include mobile and wireless technologies, healthcare such as Decision Support Systems and ICIS proceedings.

References
ABDELAAL A and ALI H (2007) A typology for community wireless DOTY DH, GLICK WH and HUBER GP (1993) Fit, equifinality, and
networks business models. In Proceedings of the 13th Americas organizational effectiveness: a test of two configurational theories.
Conference on Information Systems (HOXMEIER JA and HAYNE S, Eds), Academy of Management Journal 36(6), 1195–1250.
AIS, Keystone, CO. ELDREDGE N and CRACRAFT J (1980) Phylogenetic Patterns and the
ALDENDERFER MS and BLASHFIELD RK (1984) Cluster Analysis. Sage Evolutionary Process. Columbia University Press, New York.
Publications, Beverly Hills, CA. FIEDLER KD, GROVER V and TENG JTC (1996) An empirically derived
ANDERBERG MR (1973) Cluster Analysis for Applications. Academic Press, taxonomy of information technology structure and its relationship to
New York. organizational structure. Journal of Management Information Systems
BAILEY KD (1984) A three-level measurement model. Quality and Quantity 13(1), 9–34.
18(3), 225–245. GEIGER D, SCHULZE T, SEEDORF S, NICKERSON RC and SCHADER M
BAILEY KD (1994) Typologies and Taxonomies – An Introduction to (2011) Managing the crowd: towards a taxonomy of crowdsourcing
Classification Techniques. Sage, Thousand Oaks, CA. processes. In Proceedings of the 17th Americas Conference on Information
BAPNA R, GOES P, GUPTA A and YIWEI J (2004) User heterogeneity and its Systems (SAMBAMURTHY V and TANNIRU M, Eds), AIS, Detroit, MI.
impact on electronic auction market design: an empirical exploration. GLASS RL and VESSEY I (1995) Contemporary application-domain
MIS Quarterly 28(1), 21–43. taxonomies. IEEE Software 12(4), 63–76.
BOWKER GC and STAR SL (1999) Sorting Things Out: Classification and Its GOGUEN JA and LINDE C (1993) Techniques of requirements elicitation.
Consequences. MIT Press, Cambridge, MA. In Proceedings of the International Symposium on Requirements
DOBSON S (2004) A taxonomy for thinking about location in pervasive Engineering (FICKAS S and FINKELSTEIN A, Eds), pp 152–164, IEEE
computing. Technical Report TCD-CS-2004–05. Department of Computer Society, San Diego, CA.
Computer Science, Trinity College, Dublin. GREGOR S (2006) The nature of theory in information systems. MIS
DOGAC A, LALECI G, KABAK Y and CINGIL I (2002) Exploiting web service Quarterly 30(3), 611–642.
semantics: taxonomies vs ontologies. Bulletin of the IEEE Computer GROVE A (2003) Taxonomy. Encyclopedia of Library and Information Science
Society Technical Committee on Data Engineering. pp 2770–2777, Marcel Dekker, New York.
DOMBROVIAK KM and RAMNATH R (2007) A taxonomy of mobile and GRUBER TR (1993) A translation approach to portable ontology specifica-
pervasive applications. In Proceedings of the 2007 ACM Symposium on tions. Knowledge Acquisition 5(2), 199–220.
Applied Computing (CHO Y, WAINWRIGHT RL, HADDAD H, SHIN SY and KOO GRUNINGER M, BODENREIDER O, OLKEN F, OBRST L and YIM P (2008)
YW, Eds), pp 1609–1615, ACM, Seoul. Ontology summit 2007 – ontology, taxonomy, folksonomy: under-
DOTY DH and GLICK WH (1994) Typologies as a unique form of theory standing the distinctions. Applied Ontology 3(3), 191–200.
building: toward improved understanding and modeling. Academy of GUARINO N (1998) Formal ontology and information systems. In
Management Review 19(2), 230–251. Proceedings of FOIS ‘98 (GUARINO N, Ed), pp 3–15, IOS Press, Trento.

European Journal of Information Systems

356 A method for taxonomy development Robert C. Nickerson et al

HEINONEN K and PURA M (2006) Developing a conceptual framework for problem statement. In Proceedings of the 16th Americas Conference
mobile services. In Proceedings of the Helsinki Mobility Roundtable on Information Systems (SANTANA M, LUFTMAN JN and VINZE AS, Eds),
(JARVENPAA S, SAARINEN T and KRISTIINA V, Eds), Helsinki School of AIS, Lima.
Economics, Helsinki. NICKERSON R, VARSHNEY U, MUNTERMANN J and ISAAC H (2007) Towards
HEVNER AR and CHATTERJEE S (2010) Design Science Research in Information a taxonomy of mobile applications. In Proceedings of the 13th
Systems. Springer, New York. Americas Conference on Information Systems (HOXMEIER JA and HAYNE S,
HEVNER AR, MARCH ST, PARK J and RAM S (2004) Design science in Eds), AIS, Keystone, Co.
information systems research. MIS Quarterly 28(1), 75–105. NICKERSON R, VARSHNEY U, MUNTERMANN J and ISAAC H (2009) Taxonomy
HIRSCHHEIM RA, KLEIN HK and LYYTINEN K (1995) Information Systems development in information systems: developing a taxonomy of
Development and Data Modeling: Conceptual and Philosophical Founda- mobile applications. In Proceedings of the European Conference on
tions. Cambridge University Press, Cambridge. Information Systems (NEWELL A, WHITLEY EA, POULOUDI N, WAREHAM J and
IIVARI J (2007) A paradigmatic analysis of information systems as a design MATHIASSEN L, Eds), AIS, Verona.
science. Scandinavian Journal of Information Systems 19(2), 39–64. NYSVEEN H, PEDERSEN PE and THORBJøRNSEN H (2005) Intentions to use
KEMPER H and WOLF E (2002) Iterative process models for mobile mobile services: antecedents and cross-service comparisons. Journal of
application systems: a framework. In Proceedings of the 23rd Interna- the Academy of Marketing Science 33(3), 330–346.
tional Conference on Information Systems (MIRALLES F and VALOR J, Eds), PARSONS J and WAND Y (2008) Using cognitive principles to guide
pp 401–413, AIS, Barcelona. classification in information systems modeling. MIS Quarterly 32(4),
KRUG S, CAMPIDELLI H and NICKERSON RC (2012) A preliminary taxonomy 839–868.
for software failure impact: categorizing the impact on enterprises PINTO HS and MARTINS JP (2004) Ontologies: how can they be built?
when software fails. In Proceedings of the 18th Americas Conference on Knowledge and Information Systems 6(4), 441–464.
Information Systems (Jessup L and Valacich J, Eds), Seattle, Washington. SABHERWAL R and KING WR (1995) An empirical taxonomy of the decision-
LEEM CS, SUH HS and KIM DS (2004) A classification of mobile business making processes concerning strategic applications of information
models and its applications. Industrial Management & Data Systems systems. Journal of Management Information Systems 11(4), 177–214.
104(1), 78–87. SCHWARZ A, MEHTA M, JOHNSON N and CHIN WW (2007) Understanding
LEHMANN H and LEHNER F (2002) Making sense of mobile applications – a frameworks and reviews: a commentary to assist us in moving our field
critical note to recent approaches to their taxonomy and classification. forward by analyzing our past. The Database for Advances in
In Proceedings of the 15th Bled eCommerce Conference (GRICAR J, Ed), Information Systems 38(3), 29–50.
pp 493–507, AIS, Bled. SIMON HA (1969) The Sciences of the Artificial. The MIT Press, Cambridge,
MARCH ST and SMITH GF (1995) Design and natural science research on MA.
information technology. Decision Support Systems 15(4), 251–266. SOKAL RR and SNEATH PHA (1963) Principles of Numerical Taxonomy.
MCKINNEY JC (1966) Constructive Typology and Social Theory. Appleton- W.H. Freeman and Company, San Francisco, CA.
Centur-Crofts, New York. SOWA JF and ZACHMAN JA (1992) Extending and formalizing the
MCKNIGHT DH and CHERVANY NL (2001) What trust means in e-commerce framework for information systems architecture. IBM Systems Journal
customer relationships: an interdisciplinary conceptual typology. 31(3), 590–616.
International Journal of Electronic Commerce 6(2), 35–59. VARSHNEY U and VETTER R (2002) Mobile commerce: framework,
MILLER GA (1956) The magic number seven, plus or minus two: some applications and networking support. Mobile Networks and Applications
limits on our capacity for processing information. Psychological Review 7(3), 185–198.
101(2), 343–352. WAND Y, MONARCHI DE, PARSONS J and WOO CC (1995) Theoretical
MILLER JG and ROTH AV (1994) A taxonomy of manufacturing strategies. foundations for conceptual modeling in information systems develop-
Management Science 40(3), 285–304. ment. Decision Support Systems 15(4), 285–304.
NGAI EWT and GUNASEKARAN A (2007) A review for mobile commerce WAND Y and WEBER R (2004) Reflection: ontology in information systems.
research and applications. Decision Support Systems 43(1), 3–15. Journal of Database Management 15(2), iii–vi.
NICKERSON R (1997) A taxonomy of collaborative applications. In WEBER M (1949) The Methodology of the Social Sciences. Translated by
Proceedings of the 3rd Americas Conference on Information Systems SHILS EA and FINCH HA. Free Press, Glencoe, IL.
(GUPTA JND, Ed), pp 560–562, AIS, Indianapolis, IN. WILLIAMS K, CHATTERJEE S and ROSSI M (2008) Design of emerging digital
NICKERSON R, MUNTERMANN J and VARSHNEY U (2010) Taxonomy services: a taxonomy. European Journal of Information Systems 17(5),
development in information systems: a literature survey and 505–517.

Appendix

Papers surveyed 5. BALL N, ADAMS C and XIA W (2004) IS/IT architecture:

an integrated view and typology. In Proceedings
1. ABDELAAL A and ALI H (2007) A typology for of the 10th Americas Conference on Information
community wireless networks business models. In Systems (BULLEN C and STOHR E, Eds), pp 3753–3754,
Proceedings of the 13th Americas Conference on Informa- AIS, New York.
tion Systems (HOXMEIER JA and HAYNE S, Eds), AIS, 6. BERANEK D and HORAN T (2006) Toward an empirical
Keystone, Co. user taxonomy for personal health records systems.
2. ALTER S (1977) A taxonomy of decision support In Proceedings of the 12th Americas Conference on
systems. Sloan Management Review 19(1), 39–56. Information Systems (RODRGUEZ-ABITIA, G and IGNACIO
3. ANDERSON GA and JENSEN ED (1975) Computer AB, Eds), pp 2806–2810, AIS, Acapulco.
interconnection structures: taxonomy, characteris- 7. BITTON D, DEWITT DJ, HSAIO DK and MENON J (1984)
tics, and examples. ACM Computing Surveys 7(4), A taxonomy of parallel sorting. ACM Computing
197–213. Surveys 16(3), 287–318.
4. ARRANGA EC (2000) Cobol tools: Overview and 8. BLUM BI (1994) A taxonomy of software development
taxonomy. IEEE Software 17(2), 59–61. methods. Communications of the ACM 37(11), 82–94.

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 357

9. CARMEL E and EISENBERG J (2006) Narratives that 25. FILLEY A and ALDAG R (1978) Characteristics
software nations tell themselves: an exploration and and measurement of an organizational typology.
taxonomy. Communications of the Association for Academy of Management Journal 21(4), 578–591.
Information Systems 17(1), 851–872. 26. GILLENSON ML, SHERRELL DL and CHEN L (2000)
10. CARR P and LU Y (2007) Information technology and A taxonomy of web site traversal patterns and
knowledge worker productivity: a taxonomy of structures. Communications of the Association for In-
technology crowding. In Proceedings of the 13th formation Systems 3(1), Article 17.
Americas Conference on Information Systems (HOXMEIER 27. GREGG DG and SCOTT JE (2008) A typology of
JA and HAYNE S, Eds), AIS, Keystone, Co. complaints about Ebay sellers. Communications of
11. CASAVANT TL (1988) A taxonomy of scheduling in the ACM 51(4), 69–74.
general-purpose distributed computing systems. IEEE 28. GUMM DC (2006) Distribution dimensions in soft-
Transactions on Software Engineering 14(2), 141–154. ware development projects: a taxonomy. IEEE Soft-
12. CHANDRASEKARAN B (1983) Towards a taxonomy of ware 23(5), 45–51.
problem solving types. AI Magazine 4(1), 9–17. 29. HAMBRICK D (1983) An empirical typology of mature
13. CHIKOFSKY EJ and CROSS II JH (1990) Reverse en- industrial-product environments. Academy of Mana-
gineering and design recovery: a taxonomy. IEEE gement Journal 26(2), 213–230.
Software 7(1), 13–17. 30. HASAN H (2009) A taxonomy of modes of know-
14. CHUANG S-L and CHIEN L-F (2003) Enriching Web ledge sharing between disparate group. In Proceed-
taxonomies through subject categorization of query ings of the 13th Pacific Asia Conference on Informa-
terms from search engine logs. Decision Support tion Systems (BAPNA R and SAMBAMURTHY V, Eds), AIS,
Systems 35(1), 113–127. Hyderabad.
15. CHUANG S and CHIEN L (2005) Taxonomy genera- 31. IRANI Z and LOVE P (2001) The propaga-
tion for text segments: a practical web-based tion of technology management taxonomies for
approach. ACM Transactions on Information Systems evaluating investments in information systems.
23(4), 363–396. Journal of Management Information Systems 17(3),
16. COTTERMAN WW and KUMAR K (1989) User cube: 161–177.
a taxonomy of end users. Communications of the 32. KAFENTZIS K, APOSTOLOU D and MENTZAS G (2004)
ACM 32(11), 1313–1320. Interorganizational knowledge management sys-
17. CRAINIC TG, TOULOUSE M and GENDREAU M (1997) tems: typology and cases. In Proceedings of the
Towards a taxonomy of parallel tabu search heur- 13th European Conference on Information Systems (LEINO
istics. INFORMS Journal of Computing 9(1), 61–72. T, SAARINEN T and KLEIN S), AIS, Turku.
18. DELGADO N, GATES AQ and ROACH S (2004) A 33. KAYWORTH T, BROCATO L and WHITTEN D (2005)
taxonomy and catalog of runtime software-fault What is a chief privacy officer? An analysis based
monitoring tools. IEEE Transactions on Software En- on Mintzberg’s taxonomy of managerial roles. Com-
gineering 30(12), 859–872. munications of the Association for Information Systems
19. DENNING DE and BRANSTAD D (1996) A taxonomy for 16(6), 110–126.
key escrow encryption systems. Communications of the 34. KEARNS GS (2005) An electronic commerce strategic
ACM 39(3), 34–40. typology: insights from case studies. Information &
20. DUCASSE S and POLLET D (2009) Software architecture Management 42(7), 1023–1036.
reconstruction: a process-oriented taxonomy. IEEE 35. LANDWEHR C, BULL A, MCDERMOTT J and CHOI W (1994)
Transactions on Software Engineering 35(4), 573–591. A taxonomy of computer program security flaws.
21. DURCIKOVA A and EVERARD A (2002) An employee ACM Computing Surveys 26(3), 211–254.
typology: a knowledge management perspective. 36. LARSEN K (2003) A taxonomy of antecedents of infor-
In Proceedings of the 8th Americas Conference on mation systems success: variable analysis studies.
Information Systems (BANKER RD, CHANG H and KAO Journal of Management Information Systems 20(2),
Y-C, Eds), pp 2042–2048, AIS, Dallas, Texas. 169–246.
22. EARL M (2001) Knowledge management strategies: 37. LAUFER A (1968) A taxonomy of management theory:
toward a taxonomy. Journal of Management Informa- a preliminary framework. Academy of Management
tion Systems 18(1), 215–233. Journal 11(4), 435–442.
23. FARBEY B, LAND F and TARGETT D (1995) A taxonomy 38. LEROUGE C and GJESTLAND C (2002) A typology of
of information systems applications: the benefits data warehouse quality. In Proceedings of the
evaluation ladder. European Journal of Information 8th Americas Conference on Information Systems
Systems 4(1), 41–50. (BANKER RD, CHANG H and KAO Y-C, Eds), pp 34–37,
24. FIEDLER KD, GROVER V and TENG JTC (1996) An empi- AIS, Dallas, Texas.
rically derived taxonomy of information techno- 39. LIMONAD L and WAND Y (2009) A conceptual
logy structure and its relationship to organizational model and typology for Information Systems
structure. Journal of Management Information Systems controls. In Proceedings of the 13th Americas
13(1), 9–34. Conference on Information Systems (N ICKERSON RC

European Journal of Information Systems

358 A method for taxonomy development Robert C. Nickerson et al

and SHARDA R, Eds), pp 1–9, AIS, San Francisco, innovation typology. Decision Sciences 40(3),
California. 513–540.
40. LÓPEZ TS, RANASINGHE DC, PATKAI B and MCFARLANE D 54. PEARSON J and SHIM J (1994) An empirical investi-
(2009) Taxonomy, technology and applications gation into decision support systems capabilities:
of smart objects. Information Systems Frontiers 11(4), a proposed taxonomy. Information and Management
1–20. 27(1), 45–57.
41. LU Y and CAMPBELL S (2007) Managing the dark side 55. PLAISANT C, CARR D, and SHNEIDERMAN B (1995) Image-
of computer use at work: a typology of information browser taxonomy and guidelines for designers. IEEE
technology abuse and management strategy. In Software 12(2), 21–32.
Proceedings of the 13th Americas Conference on Informa- 56. PRUDEN HO (1973) The upward mobile, indifferent
tion Systems (HOXMEIER JA and HAYNE S, Eds), AIS, and ambivalent typology of managers. Academy of
Keystone, Co. Management Journal 16(3), 454–464.
42. MCKNIGHT D and CHERVANY N (2001) What trust 57. PUGLISI SJ, SMYTH WF and TURPIN AH (2007) A
means in E-commerce customer relationships: an taxonomy of suffix array construction algorithms.
interdisciplinary conceptual taxonomy. International ACM Computing Surveys 39(2), Article 4.
Journal of Electronic Commerce 6(2), 35–59. 58. ROBINSON SL and BENNETT RJ (1995) A typology of
43. MCKNIGHT D, CHOUDHURY V and KACMAR C (2002) deviant workplace behaviors: a multidimensional
Developing and validating trust measures in scaling study. Academy of Management Journal 38(2),
e-commerce: an integrative typology. Information 555–572.
Systems Research 13(3), 334–359. 59. SABHERWAL R and KING WR (1995) An empirical
44. MERRITT S (1985) An inverted taxonomy of sorting taxonomy of the decision-making processes con-
algorithms. Communications of the ACM 28(1), cerning strategic applications of information sys-
96–99. tems. Journal of Management Information Systems 11(4),
45. MESO P and MADEX G (2000) A complexity-based 177–214.
taxonomy of systems development methodologies. 60. S ABHERWAL R and R OBEY D (1993) An empir-
In Proceedings of the 6th Americas Conference on Informa- ical taxonomy of implementation processes
tion Systems (CHUNG M, Ed), AIS, Long Beach, CA. based on sequences of events in information
46. MILLER J and ROTH A (1994) A taxonomy of systems development. Organization Science 4(4),
manufacturing strategies. Management Science 40(3), 548–576.
285–304. 61. SELMAN AL (1994) A taxonomy of complexity classes
47. MISTILIS N and DAMBRA J (2007) A taxonomy of of functions. Journal of Computer and System Sciences
virtual information tasks and e-capability of visitor 48(2), 357–381.
information centres: an Australian case study. In 62. S ESTER A, E DER B and S CHEICHEL C (2006) Bless-
Proceedings of the 13th Americas Conference on Informa- ing or curse? A taxonomy for virtual product
tion Systems (HOXMEIER JA and HAYNE S, Eds), AIS, communities. In Proceedings of the 12th Americas
Keystone, Co. Conference on Information Systems (RODRGUEZ- ABITIA,
48. MONARCHI D and PHUR G (1992) A research typology G and I GNACIO AB, Eds), pp 4495–4503, AIS,
for object-oriented analysis and design. Communica- Acapulco.
tions of the ACM 35(9), 35–47. 63. SHENHAR AJ and BONEN Z (1997) The new taxonomy
49. NARASIPURAM M (2006) Towards a taxonomy for of systems: Toward an adaptive systems engineer-
globally distributed work. In Proceedings of the ing framework. IEEE Transactions on Systems, Man,
12th Americas Conference on Information Systems (RODR- and Cybernetics Part A: Systems and Humans 27(2),
GUEZ-ABITIA, G and IGNACIO AB, Eds), pp 867–871, AIS, 137–145.
Acapulco. 64. SON J and KIM SS (2008) Internet users’ informa-
50. NICKERSON R (1997) A taxonomy of collaborative tion privacy-protective responses: a taxonomy
applications. In Proceedings of the 3rd Americas and a nomological model. MIS Quarterly 32(3),
Conference on Information Systems (GUPTA JND, Ed), 503–529.
pp 560–562, AIS, Indianapolis. 65. SRINIVASAN V, DAVIDSON ES and TYSON GS (2004) A
51. NICKERSON R, VARSHNEY U, MUNTERMANN J and ISAAC H prefetch taxonomy. IEEE Transactions on Computers
(2007) Towards a taxonomy of mobile applications. 53(2), 126–140.
In Proceedings of the 13th Americas Conference on 66. STEFANI RT (1999) A taxonomy of sports rating
Information Systems (HOXMEIER JA and HAYNE S, Eds), systems. IEEE Transactions on Systems, Man,
AIS, Keystone, Co. and Cybernetics Part A: Systems and Humans
52. OLIVIER M and VON SOLMS S (1994) A taxonomy of 29(1), 116–120.
secure object-oriented databases. ACM Transactions on 67. TANGPONG C, MICHALISIN MD and MELCHER AJ (2008b)
Database Systems 19(1), 3–46. Toward a typology of buyer-supplier relationships:
53. PASWAN A, DSOUZA D and Z OLFAGHARIAN MA A study of the computer industry. Decision Sciences
(2009) Toward a contextually anchored service 39(3), 571–593.

European Journal of Information Systems

 A method for taxonomy development Robert C. Nickerson et al 359

68. VENUGOPAL S, BUYYA R and RAMAMOHANARAO K (2006) 71. WILLIAMS JJ and RAMAPRASAD A (1996) A taxonomy
A taxonomy of data grids for distributed data of critical success factors. European Journal of Informa-
sharing, management, and processing. ACM Comput- tion Systems 5(4), 250–260.
ing Surveys 38(1), 1–53. 72. WILLIAMS K, CHATTERJEE S and ROSSI M (2008) Design
69. VEREECKE A, VAN DIERDONCK R and DE MEYER A (2006) of emerging digital services: A taxonomy. European
A typology of plants in global manufacturing Journal of Information Systems 17(5), 505–517.
networks. Management Science 52(11), 1737–1750. 73. YOSHIOKA T, HERMAN G, YATES J and ORLIKOWSKI W
70. WANG H and WANG C (2003) Taxonomy of security (2001) Genre taxonomy: A knowledge repository
considerations in software quality. Communications of of communicative actions. ACM Transactions on
the ACM 46(6), 75–78. Information Systems 19(4), 431–456.

European Journal of Information Systems