Advanced Methods for User Evaluation in AR/VR Studies
5 likes3,155 views
Guest lecture on advanced methods of user evaluation in AR/VR studies. Given by Mark Billinghurst as part of the ARIVE lecture series hosted at the University of Otago. The lecture was given on August 26th 2021.
Ad
More Related Content
What's hot (20)
Similar to Advanced Methods for User Evaluation in AR/VR Studies (20)
Ad
More from Mark Billinghurst (20)
Ad
Recently uploaded (20)
Advanced Methods for User Evaluation in AR/VR Studies
- 1. ARIVE Advanced Methods for User Evaluation in AR/VR Studies Mark Billinghurst [email protected] August 26th 2021 ARIVE Lecture Series 2021
- 2. 2 ARIVE Expensive - $150,000+ 2 million polys/sec VGA HMD – 30 Hz Magnetic tracking Desktop VR - 1995
- 3. 3 ARIVE First Published Experiment (1995) Explore if sketch maps can be used to measure cognitive maps of Virtual Environments Hypothesis: people better oriented in VE will produce more accurate sketch maps Billinghurst, M., & Weghorst, S. (1995, March). The use of sketch maps to measure cognitive maps of virtual environments. In Proceedings Virtual Reality Annual International Symposium'95 (pp. 40-47). IEEE.
- 4. 4 ARIVE Experiment Design VR Experience n Three small simple virtual worlds n SGI Graphics + VPL HMD Hardware Between subject’s design n Each person experiences only one world n 24 – 35 subjects in each world Experiment Process 1. Training in sample world 2. Complete 24 question survey 3. 10 minutes in test world 4. Produce sketch map 5. Complete 24 question survey
- 5. 5 ARIVE Objective Measure n Map analysis n Map goodness n Object classes present n Relative object positioning Subjective Measures n 24 question survey n navigation, orientation, n interaction, presence n interface questions n 10 point Likert scale Subject comments Measures
- 6. 6 ARIVE Sample Map ZC “produce a map of the world that someone unfamiliar with the world could use to navigate around the world” Cloudlands
- 7. 7 ARIVE Results Within world correlation n Goodness and Class No. correlated with virtual world orientation and knowledge (2 worlds) Between world differences n Sign. Diff. in understanding where everything was n Sign. Diff. in placement of significant objects n Sign. Diff. in sense of dizziness in worlds
- 8. 8 ARIVE Positive Lessons n Use mixture of subjective and objective measures n Adopt existing measures from other relevant domains n Can create own experimental measures So many mistakes n Missing data n No spatial ability task n Unbalanced Likert scale n Simple experiment measures n Poor statistical analysis of data n No subject demographics reporting Lessons Learned
- 9. 9 ARIVE Collaborative AR/VR experience n See through AR displays n Exploring the role of seeing a partner’s body in a shared task Hypothesis: Seeing body will improve performance, AR better than VR Billinghurst, M., Weghorst, S., & Furness, T. (1998). Shared space: An augmented reality approach for computer supported collaborative work. Virtual Reality, 3(1), 25-36. Shared Space (1998)
- 10. 10 ARIVE Experiment Design Collaborative Task n Spotting, picking and moving objects n Simulated speech recognition n Role division: Spotter or Picker Two Factor design n Body/no body, AR/VR Conditions n RW+RB: AR - Real World + Real Body n RW: AR - Real World/No Body n VE: Virtual Environment - No Body n VE+VB: Virtual Environment + Virtual Body n VE+VB+NW: Virtual Environment + Virtual Body + No walls Virtual Body Virtual Targets
- 11. 11 ARIVE Within subject’s study n 18 pairs, aged 19-45 n No prior experience n 4 trials/condition = 20 trials Performance Time n How long to complete selection tasks Subjective Surveys n 5 Likert scale questions n Ranking of conditions Measures
- 12. 12 ARIVE Performance n No significant difference overall n Sig. Diff. bet RW+RB, VE+VB n Learning effect Subjective n Thought played better when body present n Ranked RW + RB best for performance n Ranked VE + VB best for enjoyment Results
- 13. 13 ARIVE Lessons Learned Positive Lessons n Combine Qualitative and Quantitative measures n Performance time can be a poor measure in collaborative tasks n Many factors affect performance n Use multiple subjective measures n Ranking + Likert questions Still mistakes n No user interviews n No experimenter observations n Didn’t consider learning effects in design n Poor statistical analysis (no post-hoc analysis)
- 14. 14 ARIVE Collocated Communication Behaviours Is there a difference between AR-based & screen-based FtF collaboration? Hypothesis: FtF AR produces similar behaviours to FtF non-AR Billinghurst, M., Belcher, D., Gupta, A., & Kiyokawa, K. (2003). Communication behaviors in colocated collaborative AR interfaces. International Journal of Human-Computer Interaction, 16(3), 395-423.
- 15. 15 ARIVE Experiment Design Building arranging task n Both people have half the requirements Conditions n Face to Face – FtF with real buildings n Projection – FtF with screen projection n Augmented Reality – FtF with AR buildings Face to Face Projection Augmented Reality
- 16. 16 ARIVE Measures Quantitative n Performance time n Communication Process Measures n The number and type of gestures made n The number of deictic phrases spoken n The average number of words per phrase n The number of speaker turns Qualitative n Subjective survey User comments n Post experiment interview
- 17. 17 ARIVE Results Performance time n Sig. diff. between conditions – AR slowest Communication measures n No difference in number of words/turns n Sig. Diff. in deictic phrases (FtF same as AR) n Sig. Diff. in pick gestures (FtF same as AR) Subjective measures n FtF manipulation same as AR n FtF to work with than AR/FtF Percentage Breakdown of Gestures Subject Survey Results
- 18. 18 ARIVE “working solo together”. Positive Lessons n Communication process measures valuable n Gesture, speech analysis n Collect user feedback/interviews n Stronger statistical analysis n Make observations Fewer mistakes n Surveys could be stronger n Validated surveys n Better interview analysis n Thematic analysis Lessons Learned “AR’s biggest limit was lack of peripheral vision. The interaction physically …was natural, it was just a little difficult to see.
- 19. 19 ARIVE Key Features n Training room and pit room n Physical walking n Fast, accurate, room scale tracking n Haptic feedback – feel edge of pit, walls n Strong visual and 3D audio cues Task n Carry object across pit n Walk across or walk around n Dropping virtual balls at targets in pit http://wwwx.cs.unc.edu/Research/eve/walk_exp/ UNC Pit Room (2002)
- 20. 20 ARIVE Note – from another pit experiment https://www.youtube.com/watch?v=VVAO0DkoD-8 Typical Subject Behaviour
- 21. 21 ARIVE Subjective Measures n Self report questionnaire n University College London Questionnaire (Slater 1999) n Witmer and Singer Presence Questionnaire (Witmer 1998) n ITC Sense Of Presence Inventory (Lessiter 2000) n Continuous measure n Person moves slider bar in VE depending on Presence felt Objective Measures n Behavioural n reflex/flinch measure, startle response n Physiological measures n change in heart rate, skin conductance, skin temperature Presence Slider Measuring Presence
- 22. 22 ARIVE Meehan, M., Insko, B., Whitton, M., & Brooks Jr, F. P. (2002). Physiological measures of presence in stressful virtual environments. Acm transactions on graphics (tog), 21(3), 645-652. Experiment Measures Physiological Measures n Change in heart rate n Change in skin conductance n Change in skin temperature Subjective Measures n UCL Presence questionnaire (Likert Scale) n Focus on behavioural Presence
- 23. 23 ARIVE Experiments Three experiments conducted n Effect of multiple exposures n Effect of passive haptics n Effect of framerate (10,15, 20, 30) Look at Presence correlation n Correlation between subjective scores and physiological measures Passive Haptics
- 24. 24 ARIVE Results Physiological cues n Significant change in HR in haptics/frame rate experiments n Decrease in scores with repeated exposures Presence correlation n Between HR and Presence in Frame Rate experiment n Between Skin conductance and Presence in multi-exposure
- 25. 25 ARIVE Key Lessons Learned Positive n Can use physiological cues as a process measure n Can get agreement between subjective survey results and physiological cues n Change in HR possible objective measure of Presence n Especially high Presence environments Further work n What other physiological cues could be used n Between-subjects reliability n Correlation with other Presence measures
- 26. 26 ARIVE Measuring Presence using multiple neurophysiological measures n Combining physiological and neurological signals Dey, A., Phoon, J., Saha, S., Dobbins, C., & Billinghurst, M. (2020, November). A Neurophysiological Approach for Measuring Presence in Immersive Virtual Environments. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 474-485). IEEE. Neurophysiological Measures of Presence
- 27. 27 ARIVE Dey, A., Billinghurst, M., Lindeman, R. W., & Swan, J. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 37. Meta-Review Review of 10 years of AR user studies
- 29. 29 ARIVE
- 30. 30 ARIVE Breakdown by Application Area
- 31. 31 ARIVE Breakdown by Application Area
- 32. 32 ARIVE Few AR papers have a formal experiment (~10%) Most papers use within-subjects design (73%) Most experiments in controlled environments (76%) n Lack of experimentation in real world conditions, heuristic, pilot studies Half of papers collect both Qualitative and Quantitative measures (48%) n Performance measures (76%), surveys (50%) Most papers focus on visual senses (96%) Young participants dominate (University students) (62%) n Females in minority (36%) Most use HMD (35%) or handheld displays (34%) n Handheld/mobile AR studies becoming more common Most studies are in interaction (23%), very few collaborative studies (4%) Summary
- 33. 33 ARIVE Using AR/VR to share communication cues n Gaze, gesture, head pose, body position Sharing same environment n Virtual copy of real world Collaboration between AR/VR n VR user appears in AR user’s space Piumsomboon, T., Dey, A., Ens, B., Lee, G., & Billinghurst, M. (2019). The effects of sharing awareness cues in collaborative mixed reality. Frontiers in Robotics and AI, 6, 5. Sharing: Virtual Communication Cues (2019)
- 34. 34 ARIVE Sharing Virtual Communication Cues AR/VR displays Gesture input (Leap Motion) Room scale tracking Conditions n Baseline, FoV, Head-gaze, Eye-gaze
- 35. 35 ARIVE Conditions • Baseline: In the Baseline condition, we showed only the head and hands of the collaborator in the scene. The head and hands were in all conditions • Field-of-view (FoV): We showed the FoV frustum of each collaborator to the other. This enabled collaborators to understand roughly where their partner was looking and what the other person could see at any point. • Head-gaze (FoV + Head-gaze ray): FoV frustum plus a ray originating from the user's head to identify the center of the FoV, which provided a more precise indication where the other collaborator was looking • Eye-gaze (FoV + Eye-gaze ray): In this cue, we showed a ray originating from the user's eye to show exactly where the user was looking at.
- 36. 36 ARIVE Hypotheses • H1: The Baseline condition should be the worst condition in terms of all performance metrics and behavioral observation variables. • H2: The Head-gaze and Eye-gaze conditions provide a gaze pointer, which will enable users to perform better than the FoV only condition. • H3: The Head-gaze and Eye-gaze will be favored more than Baseline condition. Not having a cue increase the collaborators' task load. • H4: The Baseline condition requires more physical movement from the collaborators as they need to look at their collaborator's avatar. • H5: The Baseline condition requires a larger distance separating the collaborators so that they could see each other's avatar.
- 37. 37 ARIVE Task Search task Two phases: n Object identification n Object placement Designed to force collaboration n Each person seeing different information
- 38. 38 ARIVE Task
- 39. 39 ARIVE
- 40. 40 ARIVE Performance Metrics n Rate of mutual gaze (objects identified/min) n Task completion time(seconds) Observed Behaviours n Number of hand gestures n Physical movement (meters) n Distance between collaborators (meters) Subjective Surveys n Usability n Social presence n Semi-structured interview Measures
- 41. 41 ARIVE Data Collected Participants n 16 pairs = 32 people n 9 women n Aged 20 – 55, average 31 years n Experience n No experience with VR (6), no experience AR (10), no HMD (7). Data collection n Objective n 4 (conditions) × 8 (trials per condition) × 16 pairs = 512 data points n Subjective n 4 (conditions) × 32 (participants) = 128 data points.
- 42. 42 ARIVE
- 43. 43 ARIVE Motion Data Map user x,y position over time
- 44. 44 ARIVE Predictions n Eye/Head pointing better than no cues n Eye/head pointing could reduce need for pointing Results n No difference in task completion time n Head-gaze/eye-gaze great mutual gaze rate n Using head-gaze greater ease of use than baseline n All cues provide higher co-presence than baseline n Pointing gestures reduced in cue conditions But n No difference between head-gaze and eye-gaze Results
- 45. 45 ARIVE Dey, A., Chatburn, A., & Billinghurst, M. (2019, March). Exploration of an EEG-based cognitively adaptive training system in virtual reality. In 2019 ieee conference on virtual reality and 3d user interfaces (vr) (pp. 220-226). IEEE. Using EEG for Adaptive VR Training Motivation n Making VR training systems adaptive in real-time to the trainee’s cognitive load to induce best level of performance gain Current VR training systems n Don’t adapt to user’s cognitive load Physiological measures n Can measure cognitive load from EEG
- 46. 46 ARIVE System Oz, O1, O2, Pz, P3, and P4
- 47. 47 ARIVE Adaption/Calibration ● Establish baseline (alpha power) ● Two sets of n(1, 2)-back tasks to calibrate own task difficulty parameters ● Measured alpha activity (task load) and calculated mean of the two tasks ● Mean → Baseline ● In experimental task ○ load > baseline → decrease level ○ load < baseline → increase level
- 48. 48 ARIVE Experimental Task Target selection n number of objects, different colors n shapes, and movement Increasing levels (0 - 20)
- 49. 49 ARIVE Experimental Task Difficulty - Low Difficulty - High
- 50. 50 ARIVE User Study ● Participants ● 14 subjects (6 women) ● 20 – 41 years old, 28 years average ● No experience with VR ● Measures ○ Response time ○ Brain activity (alpha power) 5 minutes fixed trial time
- 52. 52 ARIVE Results – Response Time Increasing levels Response Time (sec.) No difference between easiest and hardest levels
- 53. 53 ARIVE Results – Time Frequency Representation Task Load n Significant alpha synchronisation in the hardest difficulty levels of the task when compared to the easiest difficulty levels n increased cognitive effort in higher levels to sustain performance Easiest Hardest Difference
- 54. 54 ARIVE Conclusions/Future Work Conclusions n Adaptive VR training can increase the user’s cognitive load without affecting task performance n First demo of the use of real-time EEG signals to adapt the complexity of the training stimuli in a target acquisition context Future Work n Significantly increase task complexity n Can predict user performance based on the cognitive capacity n Using AR display n See real world and more distractors
- 55. 55 ARIVE Many Agents require trust n Guidance, collaboration, etc. Would you trust an agent? How can you measure trust? n Subjective/Objective measures According to AAA, 71% of surveyed Americans are afraid to ride in a fully self-driving vehicle. Understanding: Trust and Agents
- 56. 56 ARIVE Measuring Trust How to reliably measure trust? n Using physiological sensors (EEG, GSR, HRV) n Subjective measures (STS, SMEQ, NASA-TLX) Relationship between cognitive load (CL) and trust? Novelty: n Use EEG, GSR, HRV to evaluate trust at different CL n Implemented custom VR environment with virtual agent n Compare physiological, behavioral, subjective measures Gupta, K., Hajika, R., Pai, Y. S., Duenser, A., Lochner, M., & Billinghurst, M. (2020, March). Measuring human trust in a virtual assistant using physiological sensing in virtual reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 756-765). IEEE.
- 57. 57 ARIVE Experimental Task Target selection + N back memory task Agent voice guidance
- 58. 58 ARIVE 2 x 3 Expt Design Within Subject Design n 24 subjects (12 Male), 23-35 years old n All experienced with virtual assistant Two factors n Cognitive Load (Low, High) n Low = N-Back with N = 1 n High = N-Back with N = 2 n Agent Accuracy (No, Low, High) n No = No agent n Low = 50% accurate n High = 100% accurate Experiment Design
- 59. 59 ARIVE Results Physiological Measures n EEG sign. diff. in alpha band power level with CL n GSR/HRV – sign. diff. in FFT mean/peak frequency Performance n Better with more accurate agent, no effect of CL Subjective Measures n Sign. diff. in STS scores with accuracy, and CL n SMEQ had a significant effect of CL n NASA-TLX significant effect of CL and accuracy Overall n Trust for virtual agents can be measured using combo of physiological, performance, and subjective measures ”I don’t trust you anymore!!”
- 61. 61 ARIVE Pre-training (Finger Pointing) Session Start
- 62. 62 ARIVE Post-Training (Finger Pointing) Session End
- 64. 64 ARIVE
- 65. 65 ARIVE
- 66. 66 ARIVE
- 67. 67 ARIVE
- 68. 68 ARIVE New Tools New types of sensors n EEG, ECG, GSR, etc Sensors integrated into AR/VR systems n Integrated into HMDs Data processing and capture tools n iMotions, etc AR/VR Analytics tools n Cognitive3D, etc
- 69. 69 ARIVE HP Reverb G2 Omnicept Wide FOV, high resolution, best in class VR display Eye tracking, heart rate, pupillometry, and face camera
- 70. 70 ARIVE EEG attachment for AR/VR HMD 9 dry EEG electrodes https://www.next-mind.com/ NextMind
- 72. 72 ARIVE Cognitive3D Data capture and analytics for VR n Multiple sensory input (eye tracking, HR, EEG, body movement, etc) https://cognitive3d.com/
- 73. 73 ARIVE https://www.youtube.com/watch?v=tlADFAGLED4 Cognitive3D Demo
- 74. 74 ARIVE Moving Beyond Questionnaires Move data capture from post experiment to during experiment n Move from performance measures to process measures Richer types of data captured n Physiological Cues n EEG, GSR, EMG, Heart rate, etc. n Richer Behavioural Cues n Body motion, user positioning, etc. Higher level understanding n Map data to Emotion recognition, Cognitive load, etc. Use better analysis tools n Video analysis, conversation analysis, multi-modal analysis, etc.
- 75. 75 ARIVE • Types of Studies • Need for increased user studies in collaboration • More use of field studies, natural user experiences • Use a more diverse selection of participants • Evaluation measures • Need a wider range of evaluation methods • Establish correlations between objective and subject measures • Better tools • New types of physiological sensors • Develop new analytics Research Opportunities
- 76. 76 ARIVE Conclusions Most AR/VR user studies are limited n Lab based, simple qualitative/quantitative measures New opportunities for data collection n Move from post-experiment to during experiment n New sensors, analytics software Many Directions for Future Research n Data analytics n Analysis methods n Sensors n Etc..