2016 iccgis module4_thinking_aloud
- 1. APPLICATION 0F THE THINK ALOUD METHOD IN USE, USER AND USABILITY RESEARCH IN CARTOGRAPHY & GEOGRAPHIC INFORMATION SCIENCE CORNÉ VAN ELZAKKER WORKSHOP DESIGNING & CONDUCTING USER STUDIES 6TH ICC & GIS, ALBENA, BULGARIA, 15 JUNE 2016
- 2. 2 THINKING ALOUD ONE OF THE MANY METHODS OF USER RESEARCH Method Similar or Related Methods Expert- based heuristic evaluation rulesof thumb conformity inspection feature inspection, consistency inspection, standards inspection, guideline checklist cognitive walkthroughs pluralistic walkthroughs, prototyping, storyboarding, Wizard of Oz Theory- based scenario-based design personas, scenarios of use, use case, context of use, theatre secondary sources contentanalysis, competitive analysis automated evaluation automated interaction logs, unmoderated user-based methods User-based participant observation ethnographies, field observation, MILCs, journal/diary sessions, screenshot captures surveys questionnaires, entry/exit surveys, blind voting, cognitive workload assessment interviews structured interviews, semi-structured interviews, unstructured interviews, contextual inquiry focus groups supportive evaluation Delphi card sorting Q methodology, conceptmapping, aflnity diagramming, brainstorming talk/think aloud verbal protocol analysis, co-discovery study interaction study performance measurement,controlled experiments Source: Robert Roth
- 3. 3 Important aspects of usability research Requirements Analysis Evaluate early design Evaluate elaborate design User feedback Focus groups Brainstorm sessions Interviews Surveys & questionnaires Stakeholder meetings Literature review Card sorting ... Paper prototype Storyboarding Heuristic evaluation Automated evaluation Wizard of Oz Thinking aloud Eye tracking Card sorting Cognitive walkthrough .... Performance testing User satisfaction Heuristic evaluation Cognitive walkthrough Functional prototyping Critical incidents Logging user interactions .... Questionnaires User satisfaction Logging user interactions Observations Diary keeping ...
- 4. Analyse Requirements Produce Design Solutions Conceptual Design & Prototype Final Design & Prototype Interim Designs & Prototypes Identify the need for user-centred design Design satisfies requirements Evaluate Designs Testing with representative users input Business User StakeholderOrganisational Characteristics Preferences Use contexts Tasks 4 (OF GEOSPATIAL DATA PROCESSING AND DISSEMINATION SYSTEMS) Method = User-Centred Design Prototyping Requirement analysis Usability evaluation Techniques e.g.: Qualitative Focus groups Interviews Observation Thinking aloud Questionnaires Screen logging Eye tracking Performance an. Quantitative
- 5. Distinction between qualitative and quantitative research Working with experts or with (representatives of) “real” end-users Doing research in the proper context (field-based vs. laboratory based research) Distinction between methods & techniques of data collection and data analysis 5 IMPORTANT ASPECTS OF UUU RESEARCH
- 6. Interviews / Focus Groups Surveys / Questionnaires Product analysis Card sorting Interaction study Observation … COLLECTING USE, USER & USABILITY DATA ANOTHER LOOK AT CATEGORIES OF TOOLS
- 7. asking subjects to execute a particular problem-solving task and to speak out their thoughts while solving that problem / performing that task participants are asked to say out loud whatever they are looking at, thinking, doing and feeling at each moment thereafter: analyzing the resulting verbal and action protocols 7 THE THINK ALOUD METHOD
- 8. Some investigations into knowledge acquisition from existing (static) maps: Thorndyke & Stasz (1980) Kinnear & Wood a.o. (1987/1988) Kulhavy, Pridemore & Stock (1992) Crampton (1992) Heidmann & Johann (1997) Board (1997) Ungar, Blades & Spencer (1997) EARLIER APPLICATIONS OF THE THINK ALOUD METHOD IN MAP USE RESEARCH
- 9. Investigations into the use of cartographic visualization systems for geographical data exploration: McGuinness, van Wersch, Stringer (1993) MacEachren, Boscoe, Haug, Pickle (1998) See online bibliography ICA Commission on UUUI for more examples of case studies! EARLIER APPLICATIONS OF THE THINK ALOUD METHOD IN MAP USE RESEARCH
- 10. Form pairs Execute map reading activity: Locate the Google StreetView image on the map fragment provided while thinking aloud Observer takes notes or records the thinking aloud Switch roles with Map 2 After this, we will discuss your experiences 10 HANDS-ON EXPERIENCE MAP READING ACTIVITY IN PAIRS
- 11. 11 HANDS-ON EXPERIENCE MAP 1: BAILEUX
- 12. 12 HANDS-ON EXPERIENCE MAP 2: FORGES
- 13. What are your experiences with applying the think aloud method? What are the potential advantages? What are the disadvantages? What could be the problems? Questions? HANDS-ON EXPERIENCE WITH THINKING ALOUD MAP READING ACTIVITY IN PAIRS 13
- 14. Leads to valid, unique, direct and the most complete data on cognitive processes Least disturbance of the thought process No memory errors No interpretation, rationalization or explanation of thoughts by subjects No need to bring thoughts in predefined form ADVANTAGES THINK ALOUD METHOD
- 15. Very time-consuming and tedious Protocol analysis difficult and subject to interpretation by researcher Protocols may be incomplete Verbalization problems Situation may be unnatural for test participant DISADVANTAGES THINK ALOUD METHOD
- 16. Verbatim transcription of recordings: verbal protocols. Tools for qualitative data analysis e.g. Atlas.ti or MacSHAPA PROCESSING & ANALYSIS OF THINK ALOUD DATA TOOLS
- 17. Keep your research questions in mind! (but leave room to discover the unexpected ….) PROCESSING & ANALYSIS OF THINK ALOUD DATA ANALYSIS
- 18. 18 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
- 19. 19 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
- 20. 20 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
- 21. 21 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
- 22. 22 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
- 23. INTERNATIONAL INSTITUTE FOR GEO-INFORMATION SCIENCE AND EARTH OBSERVATION Corné van Elzakker The Use of Maps in the Exploration of Geographic Data
- 24. 24 Exploratory cartography maps as tools for discovery
- 25. 25 Exploration of geographic data This research project > investigation into the use of maps: the actual use of an existing cartographic display plus the cognitive process of finding, selecting, retrieving, adjusting or generating 'private' map displays
- 26. 26 User task: regional exploratory studies gaining insight into the geography of an unknown region
- 27. 27 Example of a graphical model constructed by a test person
- 28. 28 Database of the Overijssel region Available to the test persons: digital geographic (attribute and base map) data, to visualize within the ArcGIS environment existing paper maps and atlases existing digital maps (static, interactive or dynamic maps in different formats, ArcGIS ‘map documents’, maps on CD-ROM or on diskette and Web maps
- 29. 29 Searching the Overijssel database
- 30. 30 Model of regional exploratory problem- solving: map selection matrix
- 31. 31 Part C
- 32. 32 Model of regional exploratory problem-solving: map use matrix
- 33. 33 Map use matrix (upper part enlarged)
- 34. 34 Model of regional exploratory problem- solving
- 36. 36 Map use for model construction
- 37. 37 Research methodology applied questionnaire thinking aloud video recording screen transaction logging thinking aloud in retrospect analysis of verbal protocols analysis of action protocols product analysis (graphical model)
- 38. Thinking aloud is a very useful qualitative method to discover use, user and usability issues ánd requirements. Most profitable use in a mixed methods approach (e.g. including video observation, survey to establish participants’ backgrounds, screen / action logging, etc.) Combining Eye-Tracking and Thinking Aloud: demonstration Kristien Ooms 38 CONCLUSION INCORPORATE THINK ALOUD IN MIXED METHODS APPROACH
- 39. 39 ADDED VALUE OF THINKING ALOUD IN EYE-TRACKING FIND STREETVIEW IMAGE ON MAP CHIMAY
- 40. 40 EFFECTS OF THINKING ALOUD ON ET RESULTS?
- 41. Synchronization of recordings to facilitate analysis ... Further practical hints will be provided by Kristien Ooms. For now... 41 IMPLEMENTATION ISSUES THINKING ALOUD IN MIXED METHODS APPROACHES
- 42. Thinking aloud is a very useful qualitative method to discover use, user and usability issues ánd requirements. Most profitable use in a mixed methods approach. 42 CONCLUSION THINK ALOUD METHOD
- 43. APPLICATION 0F THE THINK ALOUD METHOD IN USE, USER AND USABILITY RESEARCH IN CARTOGRAPHY & GEOGRAPHIC INFORMATION SCIENCE [email protected] WORKSHOP DESIGNING & CONDUCTING USER STUDIES 6TH ICC & GIS, ALBENA, BULGARIA, 15 JUNE 2016