Evaluation in hci
Download as PPTX, PDF•1 like•3,142 views
The document discusses different methods for evaluating user interface designs, including expert evaluation techniques like heuristic evaluation and cognitive walkthroughs. It also covers user testing, which is considered more reliable than expert evaluation alone. Formative evaluation involves testing prototypes during development to identify issues, while summative evaluation assesses the final product. Both qualitative and quantitative methods are important to identify usability problems from the user's perspective.
Evaluation in hci
- 2. Nowadays users expect much more than just a usable system; they also look for a pleasing and engaging experience. This means it is even more important to carry out an evaluation. Imagine you have designed an app for teenagers to share music, gossip, and photos. You have prototyped your first design and implemented the core functionality. How would you find out whether it would appeal to them and if they will use it? You could need to evaluate it – but how?
- 3. In our human-centred approach to design, we evaluate designs right from the sketches and then as the project progresses we keep improving our prototypes and evaluate those.
- 5. Basically two types evaluation: ◦ Formative Evaluation ◦ Summative Evaluation
- 6. Formative evaluations involve evaluating a product or service during development. The goal is to make improvements in the design prior to release. This means identifying or diagnosing the problems, making and implementing recommendations, and then evaluating again. Formative usability is always done before the design has been finalized. In fact, the earlier the formative evaluation, the more impact the usability evaluations will have on the design.
- 7. Here are a few key questions you will be able answer with a formative approach: • What are the most significant usability issues preventing users from accomplishing their goals or resulting in inefficiencies? • What aspects of the product work well for the users? What do users find frustrating? • What are the most common errors or mistakes users are making?
- 8. Here are a few key questions you will be able answer with a formative approach: • What are the most significant usability issues preventing users from accomplishing their goals or resulting in inefficiencies? • What aspects of the product work well for the users? What do users find frustrating? • What are the most common errors or mistakes users are making?
- 9. • Are improvements being made from one design iteration to the next? • What usability issues can you expect to remain after the product is launched?
- 10. Summative usability is about evaluating the dish after it comes out of the oven. The usability specialist running a summative test is like a food critic who evaluates a few sample dishes at a restaurant or perhaps compares the same meal in multiple restaurants. The goal of summative usability is to evaluate how well a product or piece of functionality meets its objectives. Summative testing can also be about comparing several products to each other.
- 11. Summative usability evaluations answer these questions: • What is the overall usability of our product? • How does our product compare against the competition? • Have we made improvements from one product release to the next? . Summative testing can also be about comparing several products to each other.
- 12. Two main types ◦ Expert Evaluation Heuristic Evaluation Cognitive Walkthroughs ◦ User Testing
- 13. A simple, relatively quick and effective method of evaluation is to get an interaction design, or usability, expert to look at the system and try using it this is no substitute for getting real people to use your design expert evaluation is effective, particularly early in the design process.
- 14. Most usability engineering methods will contribute substantially to the usability of an interface … …if they are actually used.
- 15. What is it?
- 16. What is it? A discount usability engineering method - Easy (can be taught in ½ day seminar) - Fast (about a day for most evaluations) - Cheap
- 17. How does it work?
- 18. How does it work? ◦ Evaluators use a checklist of basic usability heuristics ◦ Evaluators go through an interface twice 1st pass get a feel for the flow and general scope 2nd pass refer to checklist of usability heuristics and focus on individual elements ◦ The findings of evaluators are combined and assessed
- 19. Nielsen: 10 Usability Heuristics (based on extensive empirical testing) •Visibility of system status • Match between system and the real world • User control and freedom Consistency •Error prevention • Recognition not recall • Flexibility and efficiency • Aesthetic and minimalist design • Help users diagnose and recover from errors • Help and documentation
- 20. One expert won’t do Need 3 - 5 evaluators Exact number needed depends on cost-benefit analysis
- 21. Debriefing session ◦ Evaluators rate the severity of all problems identified ◦ Use a 0 – 4, absolute scale 0 I don’t agree that this is a prob at all 1 Cosmetic prob only 2 Minor prob – low priority 3 Major prob – high priority 4 Usability catastrophe – imperative to fix
- 22. How does H.E. differ from User Testing?
- 23. How does H.E. differ from User Testing? ◦ Evaluators have checklists ◦ Evaluators are not the target users ◦ Evaluators decide on their own how they want to proceed ◦ Observer can answer evaluators’ questions about the domain or give hints for using the interface ◦ Evaluators say what they didn’t like and why; observer doesn’t interpret evaluators’ actions
- 24. What are the shortcomings of H.E.?
- 25. What are the shortcomings of H.E.? ◦ Identifies usability problems without indicating how they are to be fixed. “Ideas for appropriate redesigns have to appear magically in the heads of designers on the basis of their sheer creative powers.”
- 27. Red is used both for help messages and for error messages (consistency, match real world) “There is a problem with your order”, but no explanation or suggestions for resolution (error reporting)
- 28. No “Continue shopping" button (user control & freedom) Recalculate is very close to Clear Cart (error prevention) “Check Out” button doesn’t look like other buttons (consistency, both internal & external) Must recall and type in cart title to load (recognition not recall, error prevention, efficiency)
- 29. Expert Evaluation ◦ Heuristic Evaluation ◦ Cognitive Walkthroughs User Testing
- 31. The cognitive walkthrough was originally designed as a tool to evaluate walk-up-and-use systems like postal kiosks, automated teller machines (ATMs), and interactive exhibits in museums where users would have little or no training. However, the cognitive walkthrough has been employed successfully with more complex systems like CAD software and software development tools to understand the first experience of new users.
- 32. The cognitive walkthrough is a usability evaluation method in which one or more evaluators work through a series of tasks and ask a set of questions from the perspective of the user. Inputs: ◦ Prototype ◦ Task ◦ Sequence of actions to do the task in the prototype ◦ User analysis
- 33. For each action the evaluation asks the following questions: What is the user goal and why? Is the action obviously available? Does the action or label match the goal? Is there good feedback?
- 36. Expert evaluation is useful but there is no comparison to using the actual users for evaluation. There are many different types of user evaluations for example ◦ Formative Evaluation ◦ Field Study ◦ Controlled Experiments
- 37. Usability testing involves collecting of data using a combination of methods in a controlled setting, for example, experiments that follow basic experimental design, observation, interviews, and questionnaires. Often, usability testing is conducted in labs, although increasingly interviews and other forms of data collection are being done remotely via phone and digital communication (for instance, through Skype or Zoom) or in natural settings.
- 38. The primary goal of usability testing is to determine whether an interface is usable by the intended user population to carry out the tasks for which it was designed. This involves investigating how typical users perform on typical tasks. By typical, we mean the users for whom the system is designed (for example, teenagers, adults, and so on) and the activities that it is designed for them to be able to do (such as, purchasing the latest fashions).
- 39. As users perform the tasks, they may be recorded on video. Their interactions with the software may also be recorded, usually by logging software. User satisfaction questionnaires and interviews can also be used to elicit users’ opinions about how they liked the experience of using the system.