The Journal of Systems and Software 83 (2010) 133–142

Contents lists available at ScienceDirect

The Journal of Systems and Software

journal homepage: www.elsevier.com/locate/jss

An empirical investigation of architectural prototyping

Henrik Brbak Christensen a,*, Klaus Marius Hansen b
a
Department of Computer Science, Aarhus University, Aabogade 34, 8200 Århus N, Denmark
b
University of Iceland, Smundurgötu 2, 101 Reykjavík, Iceland

a r t i c l e i n f o a b s t r a c t

Article history: Architectural prototyping is the process of using executable code to investigate stakeholders’ software
Received 10 November 2008 architecture concerns with respect to a system under development. Previous work has established this
Received in revised form 11 June 2009 as a useful and cost-effective way of exploration and learning of the design space of a system and in
Accepted 24 July 2009
addressing issues regarding quality attributes, architectural risks, and the problem of knowledge transfer
Available online 12 August 2009
and conformance. However, the actual industrial use of architectural prototyping has not been thor-
oughly researched so far. In this article, we report from three studies of architectural prototyping in prac-
Keywords:
tice. First, we report findings from an ethnographic study of practicing software architects. Secondly, we
Software architecture
Architectural prototyping
report from a focus group on architectural prototyping involving architects from four companies. And,
Architectural analysis thirdly, we report from a survey study of 20 practicing software architects and software developers.
Architectural evaluation Our findings indicate that architectural prototyping plays an important and frequent role in resolving
Empirical software architecture problems experimentally, but less so in exploring alternative solutions. Furthermore, architectural proto-
types include end-user or business related functionality rather than purely architectural functionality.
Based on these observations we provide recommendations for effective industrial architectural
prototyping.
Ó 2009 Elsevier Inc. All rights reserved.

1. Introduction – Experimental techniques. These techniques involve some form of

demonstration to stakeholder of software architecture to inform
In practice, software architecture (Shaw and Garlan, 1996; Bass their decision making. This can be, e.g., either through demon-
et al., 2003) design is a complex process in which technical require- stration of the final system or through demonstration of aspects
ments (e.g., in the form of functional and quality requirements or of the architecture. Examples of such techniques include simula-
technology platform constraints) must to be balanced with organi- tion (Clements et al., 2002), prototyping (Bardram et al., 2004;
zational reality (e.g., stakeholder needs and concerns or organiza- Bardram et al., 2005), and scenario-based methods with explicit
tional constraints). Often this takes place within an iterative and stakeholder involvement (Kazman et al., 2000).
incremental development process where requirements and con-
straints may change frequently and fundamentally. The architect Both types of techniques are important, but arguably for devel-
has numerous techniques at his disposal that we may categorize opment processes that involve a high degree of uncertainty and
as follows: change, experimental techniques (in general) have increasing
importance since tangible demonstrations to stakeholders are
– Theoretical techniques. These techniques support software archi- needed throughout the project (Larman and Basili, 2003; Grønbk
tecture design, evaluation, or implementation through argumen- et al., 1997).
tation that tries to convince stakeholders of a specific theory For most techniques, few studies have been made of their prac-
(which could, e.g., be a specific architectural design that tries to tical (industrial) use. In this article, we focus particularly on the
resolve specific quality attribute requirements). Theoretical tech- experimental technique of architectural prototyping, providing
niques abound, examples being quality attribute scenarios (Bass new insight into the practical, industrial use of this technique. Bar-
et al., 2003), architectural patterns (Buschmann et al., 1996), dram et al. (2004) define architectural prototyping as:
view-based documentation techniques (Kruchten, 1995), and
An architectural prototype consists of a set of executables cre-
scenario-based evaluation methods (Kazman et al., 2000).
ated to investigate architectural qualities related to concerns
raised by stakeholders of a system under development. Architec-
* Corresponding author. Tel.: +45 8942 5673.
E-mail addresses: [email protected], [email protected] (H.B. Christensen), [email protected]
tural prototyping is the process of designing, building, and eval-
(K.M. Hansen). uating architectural prototypes.

0164-1212/$ - see front matter Ó 2009 Elsevier Inc. All rights reserved.
doi:10.1016/j.jss.2009.07.049
134 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142

These characteristics have been distilled from the authors’ year

long work of using architectural prototypes in various research
projects. The motivation for building prototypes was the experi-
ence that sometimes the theoretical techniques for architecture
design (theoretical in the sense defined above) provided only an
‘‘orbital view” of risks and issues but sometimes the devil was in
the detail. Building prototypes of essential architectural decisions
allowed a improved and more qualified assessment as their inher-
ent ‘‘ground view” uncovered aspects otherwise difficult or impos-
sible to foresee. As such, architectural prototyping is seen as a
viable extra design and evaluation technique to complement theo-
retical techniques.
Bardram et al. base their discussions and conclusions on case
Fig. 1. Architectural prototype ontology (adapted from Bardram et al. (2004)).
studies of the developed architectural prototypes for research pro-
jects. Thus, one may argue that the conclusions may be biased and
This definition is rendered as an ontology in Fig. 1. The ontology that conclusions may not have implications outside the research
is based on the ontology of the architectural description standard, labs. It is therefore interesting to research if architectural proto-
IEEE-Std-1471 (IEEE-Std-1471, 2000), and extended with concepts types are used in industrial practice and, if they are, to what extend
of architectural prototyping. The concepts of IEEE-Std-1471 are they have the defining characteristics put forward. The contribu-
highlighted in gray. We have refined ‘‘concerns” in our context to tion of this paper is an empirical investigation of architectural pro-
be primarily about quality attributes that are then investigated totyping in an industrial context and as such provides independent
through architectural prototypes and stress the central role of and more detailed evidence for the viability and characteristics of
the stakeholder. the technique. The independent evidence stems from three
Bardram et al. (following Floyd Floyd, 1984) classified architec- sources:
tural prototypes in terms of objective of construction1:
(i) We have conducted ethnographical studies in which
(i) Exploratory architectural prototypes are constructed in order researchers followed and observed software architects in
to explore the architecture design space. Often, several alter- their daily work.
natives are constructed, analyzed, and executed in order to (ii) We have conducted focus groups in which four architects
come up with an optimal solution to a concern. reported on their use of experimental exploration and vali-
(ii) Experimental architectural prototypes are constructed in dation of architectural solutions.
order to evaluate a specific architectural decision. Often, a (iii) We have conducted a questionnaire-based survey study in
single prototype is constructed and evaluated. which twenty software architects and developers evaluated
(iii) Evolutionary architectural prototypes are constructed as a the use of architectural prototyping in their respective
series of prototypes in which each prototype build upon companies.
the previous. Often, such a prototype leads to a skeleton sys-
tem in which little functionality is present, but where a soft-
1.1. Article outline
ware architecture design is implemented that is used as a
basis for producing a final system.2
This article is structured as follows: First, we describe the
sources of data used for our investigation: the SA@Work project
A primary contribution of the Bardram et al. paper is the iden-
involving software architects from four companies (ethnographical
tification of five characteristics that classify a prototype as architec-
and focus group study), and the continuing education course which
tural. For a prototype to be architectural, it should be:
enlists 20 experienced architects and developers (survey study)
(Section 2). Next, we present accounts of architectural prototyping
(i) constructed for exploration and learning of the architectural
based on field studies (Section 3), focus groups (Section 4), and on
design space, i.e., they are constructed to learn about the
the survey study (Section 5). These accounts are then analyzed
effect of architectural decisions largely ignoring the intent
(Section 6) and finally we discuss related work (Section 7) and con-
of the system,
clude (Section 8).
(ii) addressing issues regarding quality attributes, i.e., a main
motivation for building an architectural prototype is often
to measure (or observe) the quality implications of a 2. Sources of the study
decision,
(iii) not providing functionality per se, i.e., little or no business and The three elements of the study stem from two different and
end-user functionality is implemented, disjoint sources. The ethnographical and focus group studies draw
(iv) addressing architectural risks, i.e., the driver for architectural upon the architects in the SA@Work project while the survey study
prototyping is often an attempt to address or mitigate a risk draws upon architects from a continuing education course. There is
in architectural design, and no person overlap between the two sources. The sources are de-
(v) addressing the problem of knowledge transfer and architectural scribed in more detail below.
conformance, i.e., architectural prototypes may be used after
construction to ensure that developers learn about the archi- 2.1. The SA@Work project
tecture and that knowledge about the architecture is trans-
ferred through code. The Software Architects at Work (SA@Work) project was a one
and a half year project that started August 2007 and ended Decem-
1
In practice, of course, the construction of a specific architectural prototype may
ber 2008. It involved software architects from four Danish compa-
have several types of objectives. nies chosen among others because they represent a diversity of
2
This is, e.g., what the Rational Unified Process advocates (Kruchten, 2003). application domains. The project is described in more detail in
 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142 135

(Christensen et al., 2008a; Christensen et al., 2008b). The four com- ers in Danish IT industry. The course was taught by the authors and
panies are: organized as weekend seminars, e-learning activities, and manda-
tory exercises to complete and hand-in. As part of the mandatory
(i) Bang & Olufsen (BeO) produces high end audio products, tele- exercise program, a survey study was handed out to the students.
vision sets, and telephones. The company was founded in Section 5 describes our findings from this study.
1925, in Struer, Denmark. Since the beginning the company
has focused on creating quality products. The IT-organiza- 3. Field studies of architectural prototyping practice
tion of B&O consists of offices in Struer and Århus (Denmark)
and a subsidiary in Estonia, furthermore they work with a In the first phase of the SA@Work project we did a bottom-up,
consultancy company in India. ethnographically-inspired study of software architects. Concretely,
(ii) DSE A/S was established in 1981 in Horsens. The present each company designated a software architect that we would fol-
company is divided into two divisions, Test and Airport, with low, observe, and interact with. Specifically, we applied ‘‘participa-
respective markets. We have followed architects in the air- tory observation” in which we engaged with the software
port division that employs approximately 18 persons. The architects as necessary and did this akin to Miluk’s ‘‘rapid applied
DSE Airport Solutions division supply IT-based solutions ethnography” process (Miluk, 2006).
for Danish and international airports including ATC (Air Traf- The reason for using observations in this phase was twofold.
fic Control) and CNS (Communication, Navigation and Sur- First, we wanted to be as open as possible in our conception of ac-
veillance)/ATM (Air Traffic Management). tual architectural practice while acknowledging, though, that we
(iii) Jyske Bank (JB) is the second largest independent Danish had prior knowledge of the field and that the presence of an obser-
bank, employing some 4000 people in 119 Danish branches. ver changes what is being observed. In the analysis of observations,
The IT organization of Jyske Bank is the largest in Jutland, the presence of the observer thus has to be taken into consider-
designing, implementing, and running the systems of Jyske ation as noted in, e.g., Hammersley’s and Atkinson’s ‘‘reflexive real-
Bank in addition to processing central-government pay- ism” (Hammersley and Atkinson, 1997). Secondly, retrospective
ments and operating a payroll system for the Danish accounts such as interviews are often inaccurate because intervie-
counties. wees may not remember details about their work practice and may
(iv) Systematic Software Engineering (SSE) is Denmark’s largest also not know what is important or relevant for researchers to
privately owned software company. It was founded in know.
1985 and has since grown to over 400 employees, 50 of Fig. 2 shows a timeline of the Phase 1 interactions with software
these employed in the UK and USA. SSE is certified in process architects covering both observations and interviews. The exact
maturity at CMMI level 5. The main business areas are mis- timing and extent of the observations at the individual companies
sion critical systems for the defense and healthcare sectors. depended on the availability of architects and researchers, but we
aimed at having five days of observation at each company.
The SA@Work project was divided into two phases: One striking commonality between work practices of the archi-
tects was that the daily activities of software architects are very di-
Phase 1: Field studies of architectural practice. In this phase we verse and may change frequently according to project needs. This
have followed architects in their work for one and a half also means that specific architectural techniques were often only
to two weeks using ethnographical techniques to glimpsed, something that is also true for architectural prototyping.
observe and record actions, collaborations, artifacts, We only observed direct programming and/or execution of archi-
etc. The observations were complemented with inter- tectural prototypes in a few cases, but through the observation
views and document analysis. and followings interviews, we tentatively concluded the following:
Phase 2: Collaboration between practicing architects and
researchers. During this phase particular aspects of – Architectural prototypes are important. In all companies, critical
architectural work are identified that could be architectural decisions (such as the choice of a user interface
improved, and collaboration executed (intervention). platform or a remote procedure call middleware) had been
taken based on the construction of architectural prototypes.
Sections 3 and 4 describe observations from the two phases However, architectural prototypes complement other, theoreti-
respectively. cal, architectural techniques more than being an essential tech-
nique in its own right. It was however not clear to which extent
2.2. Continuing education course architectural prototypes were used and what were their precise
characteristic?
Our second source is architects and developers enlisted in a – Architectural prototyping needs opportunistic planning. All compa-
continuing education course, Software Architecture in Practice. The nies in our study used an iterative and incremental development
Department of Computer Science at Aarhus University has a long approach in which the activities of the software architect did not
tradition of continuing education in software engineering topics. fit well (Christensen et al., 2008b). Architectural prototyping
The course in question is a 15 ECTS course (equivalent of 1/4 study was often initiated due to uncertainties or risks in the activities
year) aimed at full time employed software architects and develop- that were part of the iterative process, meaning that architects

DSE DSE DSE

BeO BeOBeO BeOBeO BeO BeO

2007
Aug Sep Oct Nov Dec
JB JB JB
SSE SSESSE SSE

Fig. 2. Schematic overview of observations and interviews in SA@Work.

136 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142

often had to take time out of their schedule to do prototyping (or – analysis of discovery protocol behavior in various scenarios like
even do them at home). One issue that was not clear was in device out-of-range and radio signal shut down,
which phase (e.g., inception, elaboration, construction, or transi- – analysis if the selected technologies ‘‘plays well together”.
tion as in Kruchten, 2003) architectural prototyping was most
useful? As, e.g., audio sources are browsed by showing album covers the
– Architects (and developers) code architectural prototypes. In most prototype included full graphics to verify rendering quality and
cases, the software architects themselves programmed architec- performance.
tural prototypes (often because they were the experts of the
(technical) domain being investigated); at other times, archi- 4.2. Jyske Bank
tects specified the architectural prototype and a (lead) developer
would code the prototype. How the construction of architectural The architect from Jyske Bank described a large experimental
prototypes fitted into the rest of the system construction was, effort to prototype the layering aspects of a novel service oriented
however, not clear? architecture that at the moment is being introduced at the enter-
prise level to supplement and partially replace the present archi-
Among others to investigate these issues further, we initiated tecture. The layering is a traditional three-tier layering with data,
two focus groups with software architects on architectural proto- business, and presentation layers but using a service oriented
typing. The results of these are presented in the next section. approach.
The architectural aspects to be investigated were:

4. Focus groups on architectural prototyping – verification and exploration of proper layering in the architec-
ture with emphasis on achieving loose coupling and low main-
For practical reasons, architects from three of the companies tenance costs,
were present at one focus group workshop and architects from – verify that adequate performance was still attainable,
the last company were present at a separate focus group workshop. – verify the security aspects could be implemented in the layered
The architects were in both cases given a homework assignment: SOA model,
they had to review their past use of coding practices in their archi- – analyze with respect to stability and 24/7 service.
tectural work to identify between three and four instances of using
experimental, code-based, analyses or techniques in their architec- The infrastructure was prototyped along with an example appli-
tural work. Furthermore, they were asked to present one of the in- cation of using it. The example application is a Jyske Bank internal
stances in technical detail at the focus group. They were also given teaching example, KAOS, that is a course administration system.
the paper ‘‘Architectural Prototyping: An Approach for Grounding The KAOS system is well known to the entire bank’s IT staff as they
Architectural Design” by Bardram et al. (2004) as part of the initial have all gone through the same internal training.
assignment.
The first focus group seminar lasted about 6 h and the last 4.3. Systematic Software Engineering
lasted about 2 h. In both cases, it consisted of an introduction by
the authors, describing the definition and characteristics of archi- A Systematic Software Engineering software architect presented
tectural prototyping based upon the aforementioned paper. Next, a ‘‘frontier run” (Danish: ‘‘Frontløb”) which is the company’s term
the companies presented the instances of using architectural pro- for executables that explore technological or architectural issues.
totypes and questions and clarifications were asked by researchers The application explored how to add a hand-held personal digital
as well as the attending architects. The seminar ended by an anal- assistant (PDA) to the company’s Columna product which is an
ysis of the primary cases of architectural prototypes. This analysis electronic patient record (EPR) infrastructure. One functionality
was lead by the authors. The analysis went through each primary of the PDA was scanning medical bar codes.
case study from the companies and clarified whether the aspects The purpose for the prototype was:
and characteristics set forth in Bardram et al. (2004) were applica-
ble or not. – exploration of technical challenges of WiFi communication with
Below the main case study of each company is presented. PDAs using a proprietary Remote Method Invocation (RMI) sys-
tem with large data objects,
4.1. Bang and Olufsen – exploration of the Windows CE programming model,
– analysis of how to make a SOA architecture on Windows CE,
The Bang and Olufsen software architect described a ‘‘vertical – exploration of how to make web services on the Columna server.
demo” for a media library browser for a new line of audio-visual
products. This new line is based upon a product-line architecture The process actually started as two disjoint prototypes, one
that is presently being developed. The device must enable users experimenting with Windows CE and the other experimenting
to have a high quality browsing experience of a centrally stored li- with SOA. As they evolved and architectural decisions matured
brary of available audio and video streams including images and and stabilized, the two prototypes grew into a single one. The
text. architecture prototype code was after the process thrown away
Several architectural aspects were planned to be explored in the but the architectural learning lead into adopting the architecture
prototype: as the reference model for SOA for Columna.

– verification of the communication channel (the connectors) 4.4. DSE

between the various devices, including media server, connec-
tions, and the device, The software architect from DSE presented VERDI that is an ongo-
– analysis of the response times in the system under various ing project to upgrade an existing map application for the Danish
scenarios, authorities’ readiness to handle natural disasters. The upgrade was
– analysis of power management and experimentation and verifi- to provide vastly more detailed maps that put high demands on data
cation of the present architecture for this aspect, transfer and especially graphical map rendering performance.
 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142 137

Several prototypes were crafted in an exploration phase for: (i) ‘‘Architectural prototypes are important at my current
work”.
– risk analysis as it was not obvious whether to take on the project (ii) ‘‘Architectural prototypes are used frequently at my current
at all as the existing legacy application ran on 10 year old frame- work”.
works and hardware,
– assessment of whether the developed technologies/components We did not seek quantitative data on frequency (or importance)
as well as the learning outcome would enrich the company’s but rather an insight into these areas, so for both questions we
portfolio of products as the application was somewhat outside used a five-point Likert scale (with 1 being ‘‘highly disagree”, 2
the current set, being ‘‘disagree”, 3 being ‘‘neutral”, 4 being ‘‘agree”, and 5 being
– exploring Microsoft WPF (Windows Presentation Foundation) ‘‘highly agree”). The results are summarized in Fig. 3.
technologies to replace the rendering engine, In summary, three quarters of the participants (15) ‘‘agreed” or
– explore performance and optimization issues with respect to ‘‘highly agreed” to that architectural prototypes are important at
rendering speed to support usability. their current work and more than half of the participants (11)
either ‘‘agreed” or ‘‘highly agreed” to that architectural prototypes
The process included prototypes for learning and experiment- are used frequently at their current work. Conversely, only one par-
ing with reading and rendering a new vector-based map data for- ticipant ‘‘disagreed” (and none ‘‘highly disagreed”) with that are
mat and next a whole series of disjoint prototypes to assess important at their current work whereas four ‘‘disagreed” that
rendering performance using various WPF techniques. Next several architectural prototypes are used frequently (and none ‘‘highly dis-
prototypes were constructed as bit mapped maps were included agreed”). Looking at the data, we see that the scoring for frequency
and a rewrite of the rendering engine as WPF could not handle of use was consistently equal to or lower than the scoring for
the performance requirements. importance (with one exception). A tentative conclusion to this,
The four cases of architectural prototyping will be analyzed in based on the sample of 20 developers/architects, is that architec-
Section 6 together with the survey study. tural prototypes are frequently used and considered important,
but that the frequency of use does not completely match
5. A survey study of architectural prototyping importance.

The survey study exercise, completed by the 20 students on the

5.2. Classification of instances of architectural prototypes
continuing education course, consisted of background information
as well as questions related to architectural prototyping practices.
We asked the participants to discuss an architectural prototype
For background information, we asked the participants for their
from their place of work. This was done as part of a hand-in in the
number of years of industrial experience and we asked them to
course. In particular, the participants were asked to describe an in-
categorize their roles in their organization as one or more of
stance of architectural prototyping (e.g., what was the concern,
‘‘developers”, ‘‘architects”, ‘‘project managers”, and ‘‘other”. The
who were the stakeholders, how was the issue resolved). Secondly,
average number of years of experience were 10.0 years (median
they were asked to classify the architectural prototype into ‘‘exper-
8.5 years, standard deviation 6.2, least experience of 3.5 years,
imental”, ‘‘exploratory”, or ‘‘other”. Finally, they were asked to
most experience of 25 years). Eighteen described themselves as
score each of the following statements related to the characteris-
‘‘developers”, eight as ‘‘architects”, three as ‘‘project managers”,
tics of architectural prototypes as a five-point Likert scale as in Sec-
and two as ‘‘other”. All participants either categorized their role
tion 5.1. Here the text in parentheses refer to the labels in Fig. 4.
as developer or architect in their organization.
Within this settings, we performed two studies
(i) ‘‘Architectural prototypes are constructed for exploration
and learning of the architectural design space” (Learning).
(i) Importance and frequency. A questionnaire-based survey
(ii) ‘‘Architectural prototyping addresses issues regarding archi-
study of perceived importance and frequency of use of archi-
tectural quality attributes in the target system” (Quality).
tectural prototyping (see Section 5.1)
(iii) ‘‘Architectural prototypes do not provide functionality per
(ii) Classification. An essay-based survey study in which the par-
se” (No Functionality).
ticipants were asked to describe an architectural prototype
(iv) ‘‘Architectural prototypes typically address architectural
that they had been involved in building and classify it
risks” (Address Risk).
according to the objective of construction and to the charac-
(v) ‘‘Architectural prototypes address the problem of knowledge
teristics of architectural prototyping presented in Section 1.
transfer and architectural conformance” (Transfer
knowledge).
5.1. Importance and frequency of architectural prototyping
Furthermore, we classified the quality concerns of the architec-
Here the participants were presented a questionnaire in which tural prototypes according to the SEI quality attribute framework
they had to answer two statements about architectural (Bass et al., 2003). The framework was chosen since it reasonably
prototyping covers quality concerns and because the participants were very

Fig. 3. Summary of responses to questions on importance and frequency of architectural prototyping. Answers were reported as a five-point Likert scale.
138 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142

Fig. 4. Summary of responses to questions on characteristics of architectural prototyping. Answers were reported as a five-point Likert item.

familiar with it (thus hopefully leading to more precise classifica- focused on is performance (in 17 cases) followed by modifiability
tion from their side). (in 8 cases). It is no surprise that performance is the most fre-
The examples of prototypes varied widely, ranging from Web quently studied quality attribute given its general importance
service integration over an object-oriented persistence layer to and its amenability to experimental analysis. However, it is some-
experiments with online replication. The majority of the reported what more surprising that modifiability also plays a large role since
architectural prototypes were experimental (13), whereas only a it is arguably harder to experiment with.
few (3 and 4) were categorized as ‘‘exploratory” or ‘‘other”.
Summaries of responses to statements 1–5 are shown in Fig. 4 6. Analysis
where characteristics are sorted by the degree to which the partic-
ipants agreed with them. As can be seen, apart from statement (iii) An important but easily overlooked conclusion of our empirical
(‘‘Architectural prototypes do not provide functionality per se”), study is:
more than half of the participants agreed with the statements
(and with those more than three quarters of the participants Architectural prototypes are seen as important in industrial prac-
agreed except with statement (v) (‘‘Architectural prototypes ad- tice. The most prevalent type of architectural prototypes seems to
dress the problem of knowledge transfer and architectural confor- be experimental prototypes.
mance”)). Almost half of the participants (9) actually ‘‘disagreed”
or ‘‘highly disagreed” with statement (iii). We will discuss this Sec- Thus our study lends credibility to the conclusions made by Bar-
tion 6. dram et al. (2004).
Looking at the quality attributes addressed in the architectural At a more detailed level, our studies evaluated how the charac-
prototypes (Fig. 5), it is clear that the quality attribute most often teristics of architectural prototypes (see Section 1) were matched

Fig. 5. Summary of classification of the main quality attribute foci of architectural prototypes.
 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142 139

Table 1 However, all prototypes contained quite a lot of non-architec-

Characteristics of the cases and of the instances of the survey study. ‘+’ indicates that tural functionality. The Bang and Olufsen prototype media server
the given architectural prototype exhibits the identified characteristics, ‘ ’ the
opposite, while ‘(+)’ indicates that the characteristics is only partially present.
contained real album data and the device rendered graphics. The
Jyske Bank prototype contained user interface functionality, and
(1) (2) (3) (4) (5) the Columna PDA did scan real medicine bar codes. For the two
BeO: Content browser + + + (+) first prototypes, the architects stressed that they were used also
JB: SOA and application Layering + + (+) + + for demonstration to (business) stakeholders, not just for architec-
SSE: Columna PDA Scanner + + (+) + +
DSE: VERDI + + (+) +
tural evaluation. Nevertheless, they were in no way fully functional
Survey study + + (+) + + systems, but the important point here is that they did include non-
architecturally relevant code.
The DSE case is perhaps a bit special in this respect because a
against the concrete cases and the survey study. These are dis- primary concern was the risks associated with the technology up-
cussed in detail below and summarized in Table 1. date and the rendering performance. Thus one can say that the
architectural challenge indeed was the graphical functionality.
6.1. Ad 1: Exploration and learning The survey study concurs in this, the general statement being
that in order to ensure ‘‘buy-in” of architectural decisions with
This characteristic was considered essential for all SA@Work management functionality may need to be included. However,
cases by their architects as well as scores high in the survey. Con- 45% of the participants in the survey study actually agree that func-
sidering the case studies, it was perhaps especially true for the Sys- tionality per se is not part of architectural prototypes.
tematic Software Engineering Columna PDA system where the
architect and the development team faced a device (PDA) and a 6.4. Ad 4: Risks
technological platform (Windows CE) that was more or less un-
known. DSE also faced new technology for the rendering engine The case studies presented by Bang and Olufsen and Jyske Bank
in VERDI and several prototypes were made to find a WPF tech- were both examples of architectural prototypes constructed after a
nique that could handle the performance issues. Bang and Olufsen period of theoretical architectural analysis, that is, brain storming,
stressed that using RMI systems in the prototype was a relatively reviews, discussions, and other techniques that rely on abstract
new issue to be explored. That this characteristic was considered artifacts like UML diagrams, rich pictures, architect’s experience
essential concurs with the results of the survey study in which and gut-feeling, etc. Therefore many alternative architecture pro-
85% found exploration and learning to be part of architectural posals had already been weeded out before the architectural proto-
prototyping. typing processes were initiated. Therefore these prototypes have
the flavor of skeleton systems. Nevertheless both companies con-
6.2. Ad 2: Quality attributes sidered the prototype building a way to assess and control risk.
For instance, Jyske Bank mentioned their concern whether the
The software architects in the case studies all agreed that anal- new layered architecture would perform adequately when ‘‘pro-
ysis of the quality attributes were a primary driver for their archi- cessing three million rows” and the prototype was used to answer
tectural prototyping and so did all architects/developers in the this important question. Bang and Olufsen stated that the proto-
survey study (except one). This is also evident from the motivation type’s properties decided the ‘‘go/no-go” commitment for the pro-
for the prototypes as outlined in the previous section: Bang and posed architecture.
Olufsen analyzed performance metrics such as response time and The PDA prototype described by the architect from Systematic
power management; Jyske Bank explored maintainability and Software Engineering was more exploratory in that the company
availability metrics like layering and 24/7 service; Systematic Soft- had little experience with the Windows CE platform on PDA de-
ware Engineering explored performance and buildability aspects vices and therefore even less idea about how to architect SOA on
like SOA performance and Windows CE learning issues; and DSE it. Therefore the prototype building was essential both to achieve
assessed the same qualities, buildability and performance, in the confidence in programming the technological platform as well as
VERDI prototype set. From the survey study we furthermore see exploring SOA. Therefore it was a crucial risk management tool.
that performance is a dominating quality attribute concern in Finally, the DSE VERDI architectural prototyping effort was
architectural prototyping. most certainly driven by risk assessment as the company reserved
the right to decline taking on the project if the result of the proto-
6.3. Ad 3: No functionality per se typing process would indicate too high risks and too little gains
from the company’s perspective.
In our previous work we have postulated that architectural pro- The experience from the survey study is again similar, 85%
totypes do not include non-architectural functionality such as agree to this characteristic.
business logic and end-user interfaces, but only contains code cen-
tral for the quality attributes being evaluated. As an example, in 6.5. Ad 5: Knowledge transfer and conformance
two of the case studies, the architects stressed the need to see
‘‘data flowing from end-to-end”: Bang and Olufsen needed to see Also in this respect, the architects in the case study and the
album data getting from the media server to the device within developers/architects in the survey study generally agreed on the
the allowed response time while Jyske Bank needed to see data important property of communication with developers. For in-
flowing from the databases all the way ‘‘to the glass” (i.e., the user stance the Jyske Bank layering prototype was constructed using
interface) even in the face of handling three million database rows. the KAOS course management system as instantiation instead of
But for instance in the Bang and Olufsen case there was no need to the real goal, namely banking. One important property of KAOS
graphically render cover images nor even have the media server is that it is used for the internal training courses that the entire
containing real data. If response time is the issue it would suffice IT staff goes through—therefore the KAOS domain is well known
for the hand held device simply to measure round-trip times be- to all and it is very easy to introduce the new architecture on the
tween issuing a request and receiving an appropriately sized chunk simpler KAOS domain. Another important property was that a lot
of data. of template code emerged from the prototyping process that is
140 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142

used by developers to ensure conformance with the intended tives. However, they were all rejected as they did not match the
architecture. performance requirements, thus being of an experimental nature.
Systematic Software Engineering’s PDA prototype code was The present architectural prototype will now be part of an evolu-
trashed after the process and thus not used as conformance tech- tionary prototyping process to explore database issues in the
nique—however the key point of the exercise was the knowledge system.
gained for the architect and the development team. Another observation where architectural prototype usage dif-
Bang and Olufsen stressed this property less but found that the fers from the postulates in Bardram et al. (2004) is with respect
architectural prototyping process worked to increase knowledge to using architectural prototypes to explore alternatives. In sev-
transfer in the team. eral of the cases outlined by the industrial software architects,
The VERDI prototype set from DSE did not squarely fit this char- reasonably thorough but purely theoretical (in the sense outlined
acteristics. On one hand a lot of knowledge had been gained in the in Section 1) analyses where made first until a single candidate
division regarding WPF with respect to programming model and architecture was identified that stakeholders were sufficiently
performance issues but on the other hand it is less a matter of confident would match functional and architectural require-
transferring architectural knowledge from the architect to the ments. It was not until then prototyping was initiated. This is
developers. Also the final ‘‘prototype” did match the performance more in line with the concept of architectural skeletal systems as
requirements and is therefore close to a product quality finished described by Bass et al. (2003) or the usage of executable architec-
component. tural prototypes in Rational Unified Process. The data from the
survey study agrees on this since 65% of the prototype instances
6.6. Discussion were experimental.
We speculate that there is a relation between these two obser-
We conclude that four out of the five postulated characteris- vations: several of the prototypes are actually demonstrators and
tics are supported by the empirical data while data does not sup- the same prototypes are also more akin to skeletal systems than
port the fifth characteristics, no functionality per se. In the survey early explorations (Bang and Olufsen and Jyske Bank). The higher
study, less than half of the architects and developers agree to this cost of making a demonstrator requires a stronger belief that this
characteristic. In the cases presented in the focus group work- is the right architecture to reduce risks of wasted effort. Perhaps
shop, almost all had either domain or end-user functionality as this experience even lessens the architect’s tendency to architec-
part of the prototype. An extreme case was the Bang and Olufsen tural prototype purely architectural concerns.
content browser prototype that included a working user inter- One interesting issue when comparing architectural prototyp-
face having real cover and album data delivered in the system. ing to other architectural exploration and evaluation techniques
One might argue that from a purely performance assessment is cost. We did not probe the question of invested effort into the fo-
perspective this is a waste of implementation resources as per- cus group case studies nor ask for data about this in the survey
formance conclusions could be made by just transmitting realis- study. Nevertheless the discussions in the focus group workshops
tically sized byte arrays with nonsense data over the connectors as well as the perceived robustness and provided functionality of
and simply measure at what pace they arrived. This would lower the presented cases point to the fact that ‘‘substantial” effort is in-
the implementation burden as no real data should be put into the vested. This concurs with the observation that theoretical analyses
system and no GUI implemented and thereby reduce costs.3 precede the prototyping phase and only solutions that are consid-
However, one property not stressed in our previous work but ered feasible but still with some risk elements are subjected to
stressed by several architects is the importance of demonstration architectural prototyping. Moreover, it is also in line with observa-
to stakeholders, typically business decision makers. As one archi- tions by survey study participants that the actual collection and
tect stated: ‘‘‘Have a look’ is the best argument.” If decision mak- analysis of architectural prototype data may in itself be costly. Still,
ers have conflicting or impossible demands, sometimes it would be interesting to investigate this issue further to get quan-
prototypes are simply made to make it evident that they require titative data.
the impossible. Thus additional investments in implementation In the survey study, we also asked participants to outline other
effort are made to make them into working demonstrators. The characteristics of architectural prototypes than the ones that we
answers from the survey study point in the same direction and postulated. While no clear pattern emerged, the participants
add the additional point that persuasion was a characteristic of pointed to interesting problems or challenges when applying
several prototypes in that functionality was built in with the architectural prototyping. This complements our characteristics
deliberate intent of persuading stakeholders to make a specific that primarily points to positive aspects of architectural
architectural decision. prototypes.
Another classification of architectural prototypes as either A common problem in prototyping was that the organization
explorative, experimental, or evolutionary is concerned with the might decide to use the prototype as a basis for development even
architects’ focus on quality attributes (cf. Section 1). While this though it was only intended, e.g., to be used for experimentation
distinction makes sense from a theoretical stand point, the dis- with a specific quality attribute. Thus it is important to clearly state
tinction appeared blurred in industrial practice. Most of the proto- up front and agree on what the purpose of the prototype is. This
types selected by the software architects exhibited both also pertains to measurements on prototypes: if quantitative mea-
explorative as well as experimental traits and most were evolu- surements are made and goals are not set before measuring, the
tionary. The architects found the distinction less interesting from value of prototyping may be dubious.
a practical point of view as they failed to see the operational as- That only part of architectural analysis/design/evaluation is
pects. Some of the prototypes exhibited all three characteristics, considered when building an architectural prototype should also
like for instance in the DSE VERDI project in which several disjoint be taken into account. For example, if focus is on modifiability
prototypes were crafted, each exploring a particular WPF render- when building prototypes and performance is ignored, then a
ing technique, and thus of an explorative nature, testing alterna- ‘‘wrong” architectural decision may be made based on an architec-
tural prototype from the perspective of performance. And, as par-
ticipants pointed out, clearly architectural prototyping is only
3
A risk is obviously still that real-world characteristics of the scenario could be one part of architecture analysis/design/evaluation and must be
missed. Arguably such an uncertainty could also be part of architectural prototyping. complemented by other techniques.
 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142 141

7. Related work ceived ‘‘higher-than-necessary” cost may bias architects towards

not building prototypes even in situations where they would pro-
Few have written about the process of architectural prototyping vide more accurate answers to architectural dilemmas than theo-
(Kruchten, 2003; Christensen, 2005; Mårtensson et al., 2003). Both retical techniques. If this is true, even more gains can come from
Christensen (2005) and Mårtensson et al. (2003) base their exposi- architectural prototyping than reported in this work.
tion on their own experience with architectural prototyping and This speculation also hints at new directions in research. One
present a very simple framework and process that does not capture aspect of action research (Christensen et al., 2008a) is intervention
the rich use of architectural prototyping as outlined in this article. where researchers join teams of practitioners to introduce new
The Rational Unified Process (as described in Kruchten (2003)) fo- techniques. Thus one may enter a project having architectural
cuses on one type of architectural prototype, viz. an evolutionary challenges and test ways of making ‘‘cheap” architectural proto-
architectural prototype, and on a specific role of that prototype types. Another approach could be to analyze the prototype’s code
in the development process. In all cases, the analysis presented base to estimate the amount of ‘‘non-architectural” code to find
in this article extends the work by studying how architectural pro- the ratio.
totyping is done in practice.
With respect to the qualitative and empirical research approach Acknowledgements
of the SA@Work project (observations, interviews, focus groups,
surveys, and, eventually, action research Sjøberg et al., 2007) a siz- The research presented in this article has been partly funded by
able proportion of papers on software architecture argue their the ISIS Katrinebjerg competency centre, Aarhus, Denmark
claims empirically. As noted in Christensen et al. (2008a), we, (http://www.isis.alexandra.dk). We thank the companies
e.g., found that of 16 papers from WICSA 2007 (Paulish et al., and software architects that participated in the project. Further-
2007), 8 can be classified as having validated their results empiri- more, we thank the participants of the ‘‘Advanced Software Archi-
cally. This is in contrasts to (much more thorough) investigations tecture in Practice” continuing education course for participating
of the quantity of empirical validation in software engineering. in our survey study. Finally, we thank Kari Rye Schougaard who
One study, e.g., identified 12% validation through case studies (in performed a major part of the field work and Mikkel Baun Kjrg-
50 of 427 software engineering articles surveyed by Jørgensen aard and the anonymous referees whose comments improved this
and Sjøberg, 2004). Few research projects use an action research article. This paper extends results published in a paper at the Euro-
approach, though. One exception is Farenhorst et al. (2008) in pean Conference on Software Architecture 2008 (Christensen and
which observations are followed by interventions in cycles. Hansen, 2008).

8. Conclusion References

Bardram, J.E., Christensen, H.B., Hansen, K.M., 2004. Architectural prototyping: an

The first important conclusion of the presented work is perhaps approach for grounding architectural design, in: Proceedings of Fourth Working
so obvious that it may be missed, namely that it documents that IEEE/IFIP Conference on Software Architecture (WICSA4), Oslo, Norway, 2004,
architectural prototyping is being used as an indispensable tool pp. 15–24.
Bardram, J., Christensen, H., Corry, A., Hansen, K., Ingstrup, M., 2005. Exploring
in the practicing software architects’ and developers’ tool box. quality attributes using architectural prototyping, in: Quality of Software
Architectural prototypes are used as an important tool to design Architectures and Software Quality: Proceedings of the First International
and evaluate software architectures. The software architects, even Conference on the Quality of Software Architectures, QoSA 2005 and
Proceedings of the Second International Workshop on Software Quality,
experienced ones, trust the architectural answers demonstrated by SOQUA 2005, Erfurt, Germany, September, 20–22, 2005, .
executing code more than the output of brainstorming sessions, re- Bass, L., Clements, P., Kazman, R., 2003. Software Architecture in Practice, second ed.
views meetings, and logical argumentation. Addison-Wesley.
Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., Stal, M., 1996. Pattern-
Next, we conclude that the five characteristics set forward as Oriented Software Architecture: A System of Patterns. Wiley.
distinctive characteristics of architectural prototypes and setting Christensen, H., 2005. Towards an operational framework for architectural
them apart from general prototypes as defined by Floyd (1984) prototyping, in: Proceedings of the 5th Working IEEE/IFIP Conference on
Software Architecture (WICSA’05), pp. 301–302.
are generally observed in the set of architectural prototypes crafted
Christensen, H.B., Hansen, K.M., 2008. Architectural prototyping in industrial
by the participating companies and participants in the survey practice, in: Proceedings of Second European Conference on Software
study. Indeed, the architectural prototypes are used to explore Architecture, Paphos, Cyprus, pp. 196–209.
architectural designs, to learn about new architectural tactics, to Christensen, H.B., Hansen, K.M., Schougaard, K.R., 2008. Ready! Set! Go! an action
research agenda for software architecture research, in: Proceedings of Working
assess limitations and benefits of emerging technologies, to reduce IEEE/IFIP Conference on Software Architecture (WICSA) 2008, pp. 257–260.
risks of taking the wrong decisions, to experiment with the balance Christensen, H.B., Hansen, K.M., Schougaard, K.R., 2008. SA@Work – a field study of
of quality attributes in architectural proposals, and to transfer software architecture and software quality at work, in: Proceedings of Asia-
Pacific Software Engineering Conference (APSEC) 2008, Beijing, China, pp. 411–
knowledge and ensure architectural conformance in the develop- 418.
ment teams. The only characteristic that is somewhat doubtful is Clements, P., Kazman, R., Klein, M., 2002. Evaluating Software Architectures:
the claim that architectural prototypes provide little functionality Methods and Case Studies. Addison-Wesley.
Farenhorst, R., Izaks, R., Lago, P., van Vliet, H., 2008. A just-in-time architectural
besides but rather is concerned with purely architectural issues. knowledge sharing portal, in: Proceedings of the Seventh Working IEEE/IFIP
Several prototypes were effectively stakeholder demonstrators Conference on Software Architecture (WICSA 2008), pp. 125–134.
but also experimented with an architectural decision or concern. Floyd, C., 1984. A systematic look at prototyping. In: Budde, R., Kuhlenkamp, K.,
Mathiassen, L., Znllighoven, H. (Eds.), Approaches to Prototyping. Springer-
Continuing from the characteristics that we postulated and ver- Verlag, pp. 1–18.
ified, it could be of interest in future work to do a more explorative Grønbk, K., Kyng, M., Mogensen, P., 1997. Toward a cooperative experimental
study of architectural prototyping in which additional properties system development approach. In: Kyng, M., Mathiassen, L. (Eds.), Computers
and Design in Context. MIT Press, pp. 201–238.
(and characteristics) may be uncovered. This could be done by fol-
Hammersley, M., Atkinson, P., 1997. Ethnography. Principles in Practice. Routledge,
lowing a number of complete architectural prototyping processes. London & New York.
We speculate that architectural prototyping is not generally ta- The Institute of Electrical and Electronics Engineers (IEEE) Standards Board, 2000.
ken to its full potential by the software architects. The tendency to Recommended practice for architectural, ANSI/IEEE-Std-1471, September 2000.
Jørgensen, M., Sjøberg, D., 2004. Generalization and theory building in software
include demonstration quality functionality necessarily increases engineering research, in: Proceedings of the Empirical Assessment in Software
the cost of the prototypes, perhaps even substantially. This per- Engineering, pp. 29–36.
142 H.B. Christensen, K.M. Hansen / The Journal of Systems and Software 83 (2010) 133–142

Kazman, R., Klein, M., Clements, P., 2000. ATAM: Method for Architecture Sjøberg, D., Dyba, T., Jørgensen, M., 2007. The future of empirical methods in
Evaluation. Carnegie Mellon University, Software Engineering Institute. software engineering research, in: International Conference on Software
Kruchten, P., 1995. The 4 + 1 view model of architecture. IEEE Software 12 (6), 42– Engineering, pp. 358–378.
50.
Kruchten, P., 2003. The Rational Unified Process: An Introduction. Addison-Wesley
Professional. Henrik B. Christensen is an Associate Professor in the Department of Computer
Larman, C., Basili, V., 2003. Iterative and incremental development. A brief history. Science at the University of Aarhus. His research interests include Computer Science
IEEE Computer 36 (6), 47–56. education research and software architecture with a special interest in methods,
Mårtensson, F., Grahn, H., Mattsson, M., 2003. An approach for performance tools, and techniques to ensure reliable and flexible architectures. He received his
evaluation of software architectures using prototyping, in: Proceedings of the PhD in Computer Science from the University of Aarhus.
International Conference on Software Engineering and Applications (SEA 2003),
pp. 605–612.
Miluk, G., 2006. Results of a field study of cmmi for small settings using rapid Klaus Marius Hansen is a Professor of Software Engineering at University of Iceland
applied ethnography, Tech. Rep. CMU/SEI-2006-SR-001, Software Engineering (and on leave as Associate Professor at Aarhus University, Department of Computer
Institute, Carnegie Mellon, 2006. Science). His research interests are within experimental software engineering, in
Paulish, D., Gorton, I., Tyree, J., Soni, D. (Eds.), 2007. Proceedings of Working IEEE/ particular software architecture, pervasive computing, and self-managing systems.
IFIP Conference on Software Architecture (WICSA) 2007. IEEE. Klaus has a PhD in Computer Science from Aarhus University.
Shaw, M., Garlan, D., 1996. Software Architecture: Perspectives on An Emerging
Discipline. Prentice-Hall, Inc., Upper Saddle River, NJ, USA.