Knowledge-Based Systems 14 (2001) 461±465

www.elsevier.com/locate/knosys

A semiotic engineering approach to user interface design

Clarisse Sieckenius de Souza*, Simone Diniz Junqueira Barbosa, Raquel Oliveira Prates
Departamento de InformaÂtica, Semiotic Engineering Research Group, PUC-Rio, Brazil

Abstract
Designing software involves good perception, good reasoning, and a talent to express oneself effectively through programming and
interactive languages. Semiotic theories can help HCI designers increase their power to perceive, reason and communicate. By presenting
some of the results we have reached with semiotic engineering over the last few years, we suggest that the main contributions of semiotic
theory in supporting HCI design are: to provide designers with new perceptions on the process and product of HCI design; to bind together all
the stages of software development and use, giving them a unique homogeneous treatment; and to pose innovative questions that extend the
frontiers of HCI investigations. q 2001 Elsevier Science B.V. All rights reserved.
Keywords: Semiotic engineering; Interdisciplinary studies; HCI design

1. Introduction Others take a different approach and say that good

theories are the ones that allow you to understand processes
Design tasks have been challenging the value of theories and products better. They should make you aware of pitfalls
in all domains where physics, logic and mathematics cannot and false qualitative assessments. This group expects
account for the basic concepts one needs to know in order to theories to bring up new insights, and tell that ªa computer
synthesize high-quality products. Although impressive program is this; it will present features p if you have p1
theoretical scaffolding has been used to analyze a great because of p2; if you wish it to exhibit qualities such as q,
variety of human factor dimensions in artifacts, one ques- you should know that q1 and q2º. This approach places less
tion still calls for a de®nitive answer. How can this or that value on veri®cation tools and more on decision-support
theory help me make the right choices in design? information. It assumes that the quality of products is
When it comes to software design, especially in the case directly proportional to how much knowledge the designER
of interactive systems, the twofold nature of the product Ð has.
a computer program that by necessity incorporates both In a recent article [2], Philip Armour has insightfully
highly precise mathematical principles and highly subjec- characterized some aspects of the two extreme stereotypes
tive representational and communicative choices Ð expec- we have just illustrated. He calls our attention to the fact that
tations and claims about the role of theories tend to get software is not a product, but a media, where our knowledge
confusing. Some try to evade confusion by arbitrating that is encoded. If so, we can expect information-oriented
the good theories are the ones that allow you to generate true theories to play a more productive role than veri®cation-
theorems, for instance. When applied to systems design, oriented ones.
good theories should allow you to state truths of the kind So far, mainstream theoretical approaches to HCI design
ªa computer program is correct if p and qº. This group have typically drawn on cognitive perspectives. They have
expects non-mathematical theories to do something analo- had the power to explain what users do when they interact
gous, and generate truths of the following kind: ªa computer with computers, and to predict a lot of what they will or will
program is usable (or plainly good, or successful) if p1 and not do depending on the quality of the system's interface.
q2º. This should be the ®rst step to building tools to monitor The user-centered approach to system design [14] has shed
or verify that qualities expressed by p1 and q2 are incorpo- light on the fact that although user-centeredness might at
rated in a product, as a result of a good design process. ®rst glance look like a de®ning quality of any system (which
is built because of and for its users), the cognitive activity
that dominates HCI requires support that algorithms and
data structures alone fall short of providing. As a conse-
* Corresponding author.
E-mail addresses: [email protected] (C.S. de Souza), [email protected] quence, much of what has been done in HCI lately aims at
puc-rio.br (S.D.J. Barbosa), [email protected] (R.O. Prates). discovering, understanding, and supporting users in their
0950-7051/01/$ - see front matter q 2001 Elsevier Science B.V. All rights reserved.
PII: S 0950-705 1(01)00136-8
462 C.S. de Souza et al. / Knowledge-Based Systems 14 (2001) 461±465

attempt to make sense and make use of software applica- on the stage where cognitive and social theories contribute
tions. A multitude of target features have been found, and to provide insights about HCI, and encourage scientists and
handy compilations of them are offered in the form of guide- technologists to devise methods and tools to help designers
lines and heuristics for HCI design. achieve better quality in software design. In order to illus-
In a recent publication, Hollan and his co-authors [10] trate our particular way of applying semiotics to HCI design,
have pointed out that new theoretical foundations for HCI we will brie¯y sketch our own semiotic engineering
are required in view of the new challenges for communica- approach [6]. The word engineering, just as in Norman's
tion and interaction posed by current networked environ- Cognitive Engineering [15], should draw attention to the
ments. In their view, Distributed Cognition is a good synthesizing nature of the task we are trying to support
theoretical alternative because it ªseeks to understand the (an important deviation from the mostly analytical
organization of cognitive systems but, unlike traditional approaches found in traditional semiotic studies).
theories, it extends the reach of what is considered cognitive Even though the kinds of theories we draw upon to
beyond the individual to encompass interactions between achieve design HCI are quite different from those we use
people and with resources and materials in the environ- to design bridges and power plants, they all have a most
ment.º important thing in common. They are generated, inter-
This is what ethnomethodology, anthropology and sociol- preted, validated and applied by people. This means that,
ogy have brought to HCI, helping us understand the contexts to a greater or lesser extent, they are all subject to biases
where users work, the role of social structures, the impact of instilled by the effect of expectation on observation [16] and
cultural values, and the patterns of collective and individual the interplay of subjective judgment on determining objec-
behavior affecting organizational systems and technological tive goals [11].
change policies. Theoretical contributions from these disci- Unlike bridges and power plants, however, software is, in
plines have provided software designers with powerful spite of all mathematics and logic that warrant its computa-
insights and valuable knowledge that helped increase the tional status, a most subjective and arbitrary artifact. The
value of corporate systems and of computer-supported strict rules that it must abide to are only those relative to
collaborative work environments. Nevertheless, here formal symbol processing and computability. But the rela-
again, what designers got from colleagues in these other tions by which symbols gain meaning to programmers and
®elds of knowledge were more insights and new targets. users alike are in the minds of the beholders, outside the
They served to analyze the quality of existing systems, reach of algorithms and data structures.
and/or to tell designers what they should aim at, but little Designing software is thus close to writing about what
(if any) design recipes were produced. one perceives as being the case. It is a matter of good
But, are there any design recipes? What do we really gain perception, good reasoning, and a talent to express oneself
as we apply design patterns, for example? effectively. Semiotic theories can then help HCI designers
Software development is a costly process, and the price to increase their power to perceive, reason and communicate,
pay for bad design is too high. There are talented designers and semiotic engineering helps us draw a different map of
who seem to be able to get it right without knowing this territory.
anything about any theory. These are artists who excel in
their craft, and inspire us to watch them and learn. But they
are very few, and very expensive. So, we should be able to 2. A semiotic engineering approach
study these artists in the light of some theory, extract from
their performance an organized body of knowledge, and The de®ning trait of our semiotic engineering approach is
present it in a consistent and coherent theoretical discourse. that software artifacts are meta-communication artifacts.
This should eventually be transformed into a set of techni- They are one-shot messages sent from designers to users
ques that can be taught to people, leading them to be better about the range of messages users can exchange with
professionals. systems in order to achieve certain effects. This view is
It is our belief that semiotic theories are particularly well radically different from the typical user-centered design
suited to this endeavor, in that they can produce an account model of HCI, inspired by Norman's metaphor of the two
of the process of HCI designs, using models of human inter- gulfs a user must bridge to interact with systems: the execu-
pretation and expression. Moreover, they can also be used to tion and the evaluation gulf. It explicitly characterizes HCI
analyze the product of HCI design, as it is used by people in as a two-tiered communicative process involving designer-
a variety of situated contexts. This homogeneous semiotic to-user communication and user-system interaction. In our
framework used for talking about process and product view, effective HCI can only be achieved if both levels of
allows us to contrast design intentions (that emerge in the communication are successful in their particular way. In
process) with perceived meanings (that are derived from the fact, since the designer-to-user message tells the user what
product), at the very end of the line Ð the user interface. range of messages she or he can send and receive during
Therefore, like most of our colleagues have done in this interaction, and what goals and effects can be associated to
volume, we single out semiotics as a promising new player them, the success of interaction is actually dependent on the
 C.S. de Souza et al. / Knowledge-Based Systems 14 (2001) 461±465 463

react to users expectations) are consistent with and rein-

forced or explained by meanings that are explicitly
conveyed by the designer's deputy discourse (i.e. by read-
only messages present in help modules, error messages,
con®rmation dialogs, and the like). For this, a theory must
also structure the space of analysis, providing us with useful
classi®cations of signs and messages, so that correct predic-
tions can be made about the communicative effects of
choosing one or another means and mode of expression at
Fig. 1. The meta-communication process of semiotic engineering. one or another level of communication via the interface
[12].
The double articulation of HCI in semiotic engineering
success of the one-shot message. This view gives us a gives us some leverage to deal with different types of inter-
handle to deal with many issues related to apparently elusive active systems. For example, multiuser applications such as
qualities of interfaces such as intuitiveness, usability, and groupware present to HCI designers the challenge of design-
consistency, to name a few. If the user gets the designer's ing three different systems of communication: (a) the user±
top-level message about the kind of messages the system is system interaction; (b) the user±user interaction; and (c) the
prepared to interpret (how the user should phrase them, how designer-to-user communication [8,17,19]. In other words,
the user can expect the system to react, and so on), then the ®rst and foremost, the designers of such systems must have a
interface may feel intuitive, usable and consistent to the conception of the domain of application, of the group
user. In other words, these qualities in our view result structure(s) they want to support, of the mechanisms of
from a deliberate and successful communication of arbi- coordination and communication needed for productive
trary choices. They are not inherent to one or another choice collaboration, and of the direct patterns of system activation
of interface signs (widgets, images, words, layout, or dialog and system response that best ®t their design. This overall
structure). view should be conveyed directly and indirectly at all levels
The role of a semiotic theory in this perspective is then to of communication and by all instances of interfaces, a user
tell designers how they should (a) make the communicative is likely to encounter. Achieving consistency and ®nding the
choices, so that the interactive system is good, and (b) adequate means of reinforcing correct interpretations and
convey their choice to users so that this communication discouraging incorrect ones is a major design task a semiotic
does not get in the way of interaction, but serves as a secure theory should be able to support.
scaffold for productive user activity. Compared to most of Another example of how semiotic engineering can
mainstream HCI design approaches, we see that (a) is what instruct the design of special interactive systems is that of
they are after; although in our view (b) is what really extensible applications. The need to accommodate creative
increases the chances of achieving success in HCI. In Fig. activity with ®nite symbol processing mechanisms has
1 we can see how the elements borrowed from Roman motivated the development of software that users can custo-
Jakobson's characterization of a communicative process mize to their current contexts by programming new
[5] contribute to raise an HCI designer's awareness of the functions. Macro recording and scripting are the most
processes involved in semiotic engineering. The sender, or popular means to allow users to program applications,
designer, is communicating her message (about system/user although parameter con®guration, depending on the granu-
communication, its rules, purposes and possibilities) to the larity of parameters, can also achieve this effect [13].
user using the computer as medium [1] and a whole range of Semiotic engineering provides a characterization of end
visual, tactile, and aural codes. user programming as an essentially linguistic design activ-
The second distinguishing trait of our approach is that the ity, where users become designers of extensions to existing
designer is present at the interface, through what we call the applications [7,9]. The role of designers is to communicate
designer's deputy [7,18]. This abstract ®gure is in fact the not only the messages we have already described for stan-
portion of software, which carries explanations; help infor- dard interactive applications, but also the messages relative
mation, con®rmation dialogs, error messages and warnings to the syntax and semantics of another language, a meta-
about why and how some speci®c user command or language, with which users can specify new conversations
expression needs further consideration. It should convey and achievements for the original application. A special
ef®ciently direct indications of design choices and princi- means by which designers can achieve this effect is by
ples underlying the application, and thus achieve the meta- designing systems that can interpret metaphors and metony-
level communication of contents that are implicit in the mies with which users refer to new features they want to add
user±system interaction level. to applications [3,4]. They tell the system how new things
The role of a semiotic theory is now to indicate if mean- are metaphorically or metonymically related to old things,
ings implicit in the user±system interaction level (i.e. in the and thus the system learns how to react to new input. The
way users command systems to perform actions and systems role of a semiotic theory, in this case, is to illustrate how
464 C.S. de Souza et al. / Knowledge-Based Systems 14 (2001) 461±465

language can work as its own meta-language, supporting However, because it proposes to account for human inter-
expression of novel contents in terms of existing signs pretive and expressive processes underlying all activities
that are combined in speci®c rhetorical constructs the that involve representation and communication, semiotics
symbolic processor is prepared to interpret. can bind together all the stages of software development
Our semiotic engineering approach can also support the and use. There is a uniform theoretical framework that
software development cycle itself, with an evaluation can be applied from requirements elicitation to ®nal product
method that can be applied at different stages of develop- evaluation. This supports the traceability of design and
ment. It carries different messages to designers, depending implementation decisions in terms of a coherent body of
on whether the subjects performing the steps of the method knowledge, a feature other theoretical approaches have
are designers, users, or experts in semiotics. The commu- much more dif®culty to support.
nicability evaluation method [18] fully embraces our view Nevertheless, turning potential contributions into actual
that interactive systems are meta-communication artifacts. value requires critical mass and specialization. Critical mass
It serves to tell designers, in a number of ways, how well can only be achieved through coordinated research and
their message is getting across. A small set of prototypical education programs, geared speci®cally towards the inves-
utterances one might expect users to exclaim as they tigation and dissemination of new brands of Applied Semio-
encounter breakdowns and dif®culties with an application's tics, such as Computer Semiotics [1]. Specialization, in its
interface is tagged to certain patterns of interaction one turn, requires that HCI researchers and professional practi-
observes in logged sessions. Given the tasks, users were tioners embrace semiotic approaches and begin to produce
trying to perform and the tagging of utterances to break- knowledge, techniques and tools that consistently incorpo-
down situations, we can delineate a communicative pro®le rate semiotic principles and, when put to use, observably
of the interactive system. As designers examine this pro®le, lend distinct qualities to software products so achieved.
they can ponder about needs and opportunities to improve
on their message to users, or on the patterns of conversation
users can have with the system.
References
Semiotic theory, in the latter case, can provide designers
with a wealth of indications about how they are being inter- [1] P.B. Andersen, A Theory of Computer Semiotics, Cambridge Univer-
preted by users and why. Also, drawing on communication sity Press, Cambridge, 1990.
techniques well known to theorists of human communica- [2] P.G. Armour, The Business of Software: the case for a new business
tion, semiotics can present designers with alternative means model, Commun. ACM 43 (8) (2000) 19±22.
and modes of expression, in view of the effects they want to [3] S.D.J. Barbosa, C.S. de Souza, Extending software through metaphors
and metonymies, Knowledge-based Systems 14 (1±2) (2001) pp. 15±
achieve. Experimental data may even be used to build case
27.
bases for supporting other design tasks and helping identify [4] S.D.J. Barbosa, ProgramacËaÄo via Interface, Doctoral Thesis, Depar-
patterns that can eventually be part of computer-aided user tamento de InformaÂtica, PontifõÂcia Universidade CatoÂlica do Rio de
interface design environments. Janeiro, RJ, Brasil, May 1999.
[5] Jacobson, R. (1960) Linguistics and Poetics. in T.A. Sebeok (ed.)
Style in Language. New York. John Wiley and Sons. pp. 350±377.
[6] C.S. de Souza, The semiotic engineering of user interface languages,
3. Conclusions Int. J. Man±Machine Studies 39 (1993) 753±773.
[7] C.S. de Souza, Supporting end-user programming with explanatory
The conclusions we have reached in our own pursuit of discourse, Proceedings of ISAS'97 Ð Intelligent Systems and Semio-
how semiotic approaches can contribute to user interface tics'97 Ð A Learning Perspective, 22±25 September, NIST,
design can be simply stated. First, we believe that the role Gaithersburg, USA, 1997 pp. 461±466.
[8] C.K.V. Cunha, C.S. de Souza, V.S.T.D.B. Quental, D. Schwabe, A
of semiotic theory in supporting HCI design is not one of
model for extensible web-based information-intensive task-oriented
discriminating true from false or correct from incorrect, systems, in: M. de Sharon, Y. Waern, G. Cockton (Eds.), People and
predominantly, but rather that of providing designers with Computers XIV Ð Usability or Else!, HCI2000 Ð 14th Annual
new perceptions on the process and product of HCI designs. Conference of the British HCI Group, The British Computing Society,
What can be gained by such perceptions is exempli®ed by University of Sunderland (UK), 5±8 SeptemberSpringer, London,
our semiotic engineering approach and the implications of a 2000.
[9] C.S. de Souza, S.D.J. Barbosa, S.R.P. da Silva, Semiotic engineering
two-fold communicative structuring of our HCI model. principles for evaluating end-user programming environments, Interact-
Causes and consequences of design choices, compared to ing with Computers 13 (4) (2001) pp. 467±495.
mainstream user-centered models, are stated in a substan- [10] J. Hollan, E. Hutchins, D. Kirsh, Distributed cognition: toward a new
tially different perspective, which engenders different foundations for human-computer interaction research, ACM Trans.
problem formulations and, by the same token, different solu- Computer±Human Interact. 7 (2) (2000) 174±196.
[11] I. Kant, Critique of Pure Reason, St. Martin Press, New York, 1965.
tion spaces.
[12] J.C. Leite. Modelos e Formalismos para a Engenharia SemioÂtica de
Second, we do not believe that semiotic theory can, by Interfaces de UsuaÂrio, Doctoral Thesis, Departamento de InformaÂtica,
itself alone, account for all the results and insights brought PontifõÂcia Universidade CatoÂlica do Rio de Janeiro, RJ, Brasil, Octo-
forth to HCI by cognitive and social theories. It cannot. ber 1998.
 C.S. de Souza et al. / Knowledge-Based Systems 14 (2001) 461±465 465

[13] B. Nardi, A Small Matter of Programming, Mass. MIT Press, CRIWG'98 Ð IV International Workshop On Groupware, ArmacËaÄo
Cambridge, 1993. de BuÂzios, Brazil, 1998 pp. 53±67.
[14] D.A. Norman, S.W. Draper, User Centered System Design, Lawrence [18] R.O. Prates, C.S. de Souza, S.D.J.B. Barbosa, A method for evaluat-
Erlbaum Associates, Hillsdale, NJ, 1986. ing the communicability of user interfaces, ACM Interact. (Jan-Feb
[15] D.A. Norman, Cognitive Engineering, in: D.A. Norman, S.W. Draper 2000) 31±38.
(Eds.), User Centered System Design, Lawrence Erlbaum Associates, [19] R.O. Prates. A Engenharia SemioÂtica de Linguagens de Interfaces
Hillsdale, NJ, 1986, pp. 31±61. Multi-UsuaÂrio, Doctoral Thesis, Departamento de InformaÂtica, Ponti-
[16] K. Popper, Objective Knowledge, Clarendon Press, London, 1972. fõÂcia Universidade CatoÂlica do Rio de Janeiro, RJ, Brasil, October
[17] R.O. Prates, C.S. de Souza, Towards a semiotic environment for 1998.
supporting the development of multi-user interfaces, Proceedings of