Evaluation Techniques and Universal Design

Evaluation Techniques
Evaluation Tests usability and functionality of system Occurs in laboratory, field and/or in collaboration with users Evaluates both design and implementation

Goals of Evaluation
Assess extent of system functionality Assess effect of interface on user Identify specific problems

Evaluating Designs
Cognitive Walkthrough Heuristic Evaluation Review-based evaluation

Cognitive Walkthrough
Proposed by Polson et al. Evaluates design on how well it supports user in learning task Usually performed by expert in cognitive psychology Expert walks through design to identify potential problems using psychological principles For each task walkthrough considers What impact will interaction have on user? What cognitive processes are required? What learning problems may occur? Analysis focuses on goals and knowledge: does the design lead the user to generate the correct goals?

Heuristic Evaluation
Proposed by Nielsen and Molich. Usability criteria (heuristics) are identified Design examined by experts to see if these are violated Example heuristics System behaviour is predictable System behaviour is consistent Feedback is provided Heuristic evaluation `debugs' design. Review-based evaluation Results from the literature used to support or refute parts of design. Care needed to ensure results are transferable to new design. Model-based evaluation

Evaluating Implementations
Requires an artefact: simulation, prototype, full implementation

Experimental evaluation
Controlled evaluation of specific aspects of interactive behaviour
Evaluator chooses hypothesis to be tested

A number of experimental conditions are considered which differ only in the value of some controlled variable.
Changes in behavioural measure are attributed to different conditions

Experimental factors
Subjects
who representative, sufficient sample

Variables
things to modify and measure

Hypothesis
what youd like to show

Experimental design
how you are going to do it

Variables
Independent variable (IV)
characteristic changed to produce different conditions e.g. interface style, number of menu items

Dependent variable (DV)

Characteristics measured in the experiment e.g. time taken, number of errors.

Hypothesis
Prediction of outcome
Framed in terms of IV and DV e.g. error rate will increase as font size decreases

Null hypothesis:
States no difference between conditions Aim is to disprove this

e.g. null hyp. = no change with font size

Experimental studies on groups

More difficult than single-user experiments
Problems with:
subject groups choice of task data gathering analysis

Subject groups
larger number of subjects more expensive
longer time to `settle down even more variation! difficult to timetable so often only three or four groups

The task must encourage cooperation perhaps involve multiple channels

Data gathering several video cameras-direct logging of application Problems-synchronisation one solution-record from each perspective
Analysis N.B. vast variation between groups look at interactions between group and media controlled experiments may `waste' resources!

Observational Methods
Think Aloud Cooperative evaluation Post-task walkthroughs

Think Aloud
User observed performing task User asked to describe what he is doing and why, what he thinks is happening etc. Advantages
Simplicity - requires little expertise Can provide useful insight Can show how system is actually use

Disadvantages
Subjective Selective Act of describing may alter task performance

Cooperative evaluation
Variation on think aloud User collaborates in evaluation Both user and evaluator can ask each other questions throughout
Additional advantages
Less constrained and easier to use User is encouraged to criticize system Clarification possible

Post-task walkthroughs
Transcript played back to participant for comment
Immediately fresh in mind Delayed evaluator has time to identify questions

Useful to identify reasons for actions and alternatives considered Necessary in cases where think aloud is not possible

Query Techniques
Interviews Questionnaires

Interviews
Analyst questions user on one-to -one basis usually based on prepared questions Informal, subjective and relatively cheap Advantages
Can be varied to suit context Issues can be explored more fully Can elicit user views and identify unanticipated problems

Disadvantages
Very subjective Time consuming

Questionnaires
Set of fixed questions given to users Advantages Quick and reaches large user group Can be analyzed more rigorously Disadvantages Less flexible Less probing Need careful design What information is required? How are answers to be analyzed? Styles of question Open-ended Scalar Multi-choice Ranked

Physiological methods
Eye tracking Physiological measurement

Eye tracking
Head or desk mounted equipment tracks the position of the eye Eye movement reflects the amount of cognitive processing a display requires Measurements include
Fixations: eye maintains stable position. Number and duration indicate level of difficulty with display Saccades: rapid eye movement from one point of interest to another Scan paths: moving straight to a target with a short fixation at the target is optimal

Physiological measurements
Emotional response linked to physical changes These may help determine a users reaction to an interface Measurements include:
Heart activity, including blood pressure, volume and pulse. Activity of sweat glands: Galvanic Skin Response (GSR) Electrical activity in muscle: electromyogram (EMG) Electrical activity in brain: electroencephalogram (EEG)

Some difficulty in interpreting these physiological responses - more research needed

Choosing an Evaluation Method

when in process:
how objective: type of measures:

design vs. implementation

subjective vs. objective qualitative vs. quantitative

level of information:
level of interference: resources available:

high level vs. low level

obtrusive vs. unobtrusive time, subjects, equipment, expertise

universal design

Universal design principles

Equitable (for all) use Flexibility in use Simple and intuitive to use Perceptible information Tolerance for error Low physical effort Size and space for approach and use

Multi-Sensory Systems More than one sensory channel in interaction e.g. sounds, text, hypertext, animation, video, gestures, vision Used in a range of applications: particularly good for users with special needs, and virtual reality

Usable Senses
The 5 senses (sight, sound, touch, taste and smell) are used by us every day
each is important on its own together, they provide a fuller interaction with the natural world

Computers rarely offer such a rich interaction Can we use all the available senses?
ideally, yes practically no

We can use

sight sound touch (sometimes)

We cannot (yet) use taste smell

Multi-modal vs. Multi-media

Multi-modal systems
use more than one sense (or mode ) of interaction
e.g. visual and aural (hearing) senses: a text processor may speak the words as well as echoing them to the screen

Multi-media systems
use a number of different media to communicate information
e.g. a computer-based teaching system:may use video, animation, text and still images: different media all using the visual mode of interaction; may also use sounds, both speech and non-speech: two more media, now using a different mode

Speech
Human beings have a great and natural mastery of speech makes it difficult to appreciate the complexities but its an easy medium for communication Structure of Speech phonemes basic atomic units sound slightly different depending on the context they are in. e.g. the sound k in skill, kit, work, etc allophones all the sounds in the language morphemes smallest unit of language that has meaning. The smallest component of a word. eg unbreakable-3 morphemes un-break-able

Speech
Other terminology: prosody alteration in tone and quality variations in emphasis, stress, pauses and pitch impart more meaning to sentences. co-articulation the effect of context on the sound syntax structure of sentences semantics meaning of sentences Speech Recognition-enter speech data when users hands are already occupied e.g. driving, manufacturing for users with physical disabilities Problems Different people speak differently The syntax of semantically similar sentences may vary. Background noises can interfere.

Speech Synthesis
The generation of speech Useful for natural and familiar way of receiving information Problem
- needs headphones - creates noise in the workplace

Examples: screen readers-read the textual display to the user utilised by visually
impaired people

warning signals-spoken information sometimes presented to pilots whose

visual and haptic skills are already fully occupied

Non-Speech Sounds
boings, bangs, squeaks, clicks etc. commonly used for warnings and alarms Evidence to show they are useful fewer typing mistakes with key clicks video games harder without sound Language independent, unlike speech

Auditory Icons
Everyday sounds mapped to events in computer system Use natural sounds to represent different types of object or action Natural sounds have associated semantics which can be mapped onto similar meanings in the interaction e.g. throwing something away ~ the sound of smashing glass Problem: not all things have associated meanings SonicFinder for the Macintosh items and actions on the desktop have associated sounds folders have a papery noise moving files dragging sound copying sound of a liquid being poured into a receptacle rising pitch indicates the progress of the copy big files have louder sound than smaller ones

Earcons
An earcon is a brief, distinctive sound used to represent a specific event. They are a common feature of computer operating systems, where a warning or an error message is accompanied by a distinctive tone or combination of tones. Other examples would include the sound that indicates an incoming message in a chat utility. Synthetic sounds used to convey information Structured combinations of notes (motives ) represent actions and objects.

touch
haptic interaction
tactile sensation; vibrations on the skin

information on shape, texture, resistance, temperature, comparative spatial factors example technologies
electronic braille displays

Handwriting recognition
Handwriting is another communication mechanism which we are used in day-to-day life Handwriting consists of complex strokes and spaces Captured by digitising tablet strokes transformed to sequence of dots large tablets available suitable for digitising maps and technical drawings smaller devices, some incorporating thin screens to display the information PDAs such as Palm Pilot tablet PCs

Gesture
Eg for sign language enhance communication between signing and non-signing users It is user dependent

 visual impairment-screen readers, SonicFinder hearing impairment-text communication, gesture, captions physical impairment-speech I/O, eyegaze, gesture, predictive systems (e.g. Reactive keyboard) speech impairment-speech synthesis, text communication Dyslexia=problem to read, spell, etc-speech input, output Autism-communication, education PLUS age groups-older people e.g. disability aids, memory aids, communication tools to prevent social isolation children e.g. appropriate input/output devices, involvement in design process cultural differences-influence of nationality, generation, gender, race, sexuality, class, religion, political persuasion etc. on interpretation of interface features e.g. interpretation and acceptability of language, cultural symbols, gesture and colour

Users with disabilities