Comp4010 lecture11 VR Applications
1 like1,005 views
The lecture discusses interaction techniques in virtual reality (VR), covering tasks such as object selection, manipulation, navigation, and system control, emphasizing various methods like ray-casting and world-in-miniature. Additionally, it addresses the importance of user experience design in VR applications across domains such as education, medicine, and entertainment, outlining evaluation methods and user needs. Key takeaways include the necessity for immersive experiences and proper interaction design to enhance usability and learning outcomes in virtual environments.
Comp4010 lecture11 VR Applications
- 1. VR APPLICATIONS COMP 4010 Lecture Eleven Mark Billinghurst October 19th 2021 [email protected]
- 3. How can we Interact in VR? • How can VR devices create a natural user experience?
- 4. Universal 3D Interaction Tasks in VR • Object Interaction • Selection: Picking object(s) from a set • Manipulation: Modifying object properties • Navigation • Travel: motor component of viewpoint motion • Wayfinding: cognitive component; decision-making • System control • Issuing a command to change system state or mode
- 5. Selection and Manipulation • Selection: • specifying one or more objects from a set • Manipulation: • modifying object properties • position, orientation, scale, shape, color, texture, behavior, etc.
- 6. Common Selection Techniques •Simple virtual hand •Ray-casting •Occlusion •Go-go (arm-extension)
- 7. Ray-casting technique • “Laser pointer” attached to virtual hand • First object intersected by ray may be selected • User only needs to control 2 DOFs • Proven to perform well for remote selection • Variants: • Cone casting • Snap-to-object rays
- 8. Common Manipulation Techniques •Simple virtual hand •HOMER •Scaled-world grab •World-in-miniature
- 9. World-in-miniature (WIM) technique • “Dollhouse” world held in user’s hand • Miniature objects can be manipulated directly • Moving miniature objects affects full-scale objects • Can also be used for navigation Stoakley, R., Conway, M., & Pausch, R. (1995). Virtual Reality on a WIM: Interactive Worlds in Miniature. Proceedings of CHI: Human Factors in Computing Systems, 265-272, and Pausch, R., Burnette, T., Brockway, D., & Weiblen, M. (1995). Navigation and Locomotion in Virtual Worlds via Flight into Hand-Held Miniatures. Proceedings of ACM SIGGRAPH, 399-400.
- 10. Navigation • How we move from place to place within an environment • The combination of travel with wayfinding • Wayfinding: cognitive component of navigation • Travel: motor component of navigation • Travel without wayfinding: "exploring", "wandering”
- 11. Types of Travel • Exploration • No explicit goal for the movement • Search • Moving to specific target location • Naïve – target position not known • Primed – position of target known • Maneuvering • Short, precise movements changing viewpoint
- 12. Gaze Directed Steering • Move in direction that you are looking • Very intuitive, natural navigation • Can be used on simple HMDs (e.g. Google Cardboard) • But: Can’t look in different direction while moving
- 13. TelePortation • Use controller to select end point • Usable with 3DOF contoller • Jump to a fixed point in VR • Discrete motion can be confusing/cause sickness
- 14. Redirected Walking • Address problem of limited walking space • Warp VR graphics view of space • Create illusion of walking straight, while walking in circles Razzaque, S., Kohn, Z., & Whitton, M. C. (2001, September). Redirected walking. In Proceedings of EUROGRAPHICS (Vol. 9, pp. 105-106).
- 15. Wayfinding – Making Cognitive Maps • Goal of Wayfinding is to build Mental Model (Cognitive Map) • Types of spatial knowledge in a mental model • landmark knowledge • procedural knowledge (sequence of actions required to follow a path) • map-like (topological) knowledge • Creating a mental model • systematic study of a map • exploration of the real space • exploration of a copy of the real space • Problem: Sometimes perceptual judgments are incorrect within a virtual environment • e.g. users wearing a HMD often underestimate dimensions of space, possibly caused by limited field of view
- 16. Designing VE to Support Wayfinding • Provide Landmarks • Any obvious, distinct and non-mobile object can serve as a landmark • A good landmark can be seen from several locations (e.g. tall) • Audio beacons can also serve as landmarks • Use Maps • Copy real world maps • Ego-centric vs. Exocentric map cues • World in Miniature • Map based navigation
- 17. System Control • Issuing a command to change system state or mode • Examples • Launching application • Changing system settings • Opening a file • Etc. • Key points • Make commands visible to user • Support easy selection
- 18. 2D Menus in VR • Many examples of 2D GUI and floating menus in VR Nested Pie Menu 2D Menu in VR CAVE
- 19. How Can we Design Useful VR? • Designing VR experiences that meet real needs
- 20. The Interaction Design Process Evaluate (Re)Design Identify needs/ establish requirements Build an interactive version Final Product Develop alternative prototypes/concepts and compare them And iterate, iterate, iterate....
- 21. Methods for Identifying User Needs Learn from people Learn from analogous settings Learn from Experts Immersive yourself in context
- 22. Is VR the Best Solution? • Not every problem can be solved by VR.. • Problems Ideal for Virtual Reality, have: • visual elements • 3D spatial interaction • physical manipulation • procedural learning • Problems Not ideal for Virtual Reality, have: • heavy reading, text editing • many non-visual elements • need for connection with real world • need for tactile, haptic, olfaction feedback
- 23. Typical VR Interface Metaphors • Direct Manipulation • Reach out and directly grab objects • Ray Casting • Select objects through ray from head/hand • Vehicle Movement • Move through VR environment through vehicle movement
- 24. Affordances in VR • Design interface objects to show how they are used • Use visual cues to show possible affordances • Perceived affordances should match actual affordances • Good cognitive model - map object behavior to expected Familiar objects in Job Simulator Object shape shows how to pick up
- 25. UX Guidelines for VR • The Four Cores of UX Design for VR • Make interface Interactive and Reactive • Design for Comfort and Ease • Use usable Text and Image Scale • Include position audio and 3D sound https://www.dtelepathy.com/blog/philosophy/ux-guide-designing-virtual-reality-experiences
- 26. Cardboard Design Lab • Mobile VR App providing examples of best practice VR designs and user interaction (iOS, Play app stores)
- 27. What is Evaluation? •Evaluation is concerned with gathering data about the usability of a design or product by a specified group of users for a particular activity within a specified environment or work context
- 28. Four Evaluation Paradigms •‘quick and dirty’ •usability testing (lab studies) •field studies •predictive evaluation
- 29. Quick and Dirty • ‘quick & dirty’ evaluation: informal feedback from users or consultants to confirm that their ideas are in-line with users’ needs and are liked. • Quick & dirty evaluations are done any time. • Emphasis is on fast input to the design process rather than carefully documented findings.
- 30. Usability Testing • Recording typical users’ performance on typical tasks in controlled settings. • As the users perform tasks they are watched & recorded on video & their inputs are logged. • User data is used to calculate performance times, errors & help determine system usability • User satisfaction questionnaires & interviews are used to elicit users’ opinions.
- 31. Field/Ethnographic Studies • Field studies are done in natural settings • The aim is to understand what users do naturally and how technology impacts them. • In product design field studies can be used to: - identify opportunities for new technology - determine design requirements - decide how to introduce new technology - evaluate technology in use.
- 32. Predictive Evaluation • Experts apply their knowledge of typical users, often guided by heuristics, to predict usability problems. • Can involve theoretically based models. • A key feature of predictive evaluation is that users need not be present • Relatively quick and inexpensive
- 33. Pilot Studies • A small trial run of the main study. • Can identify majority of issues with interface design • Pilot studies check: - that the evaluation plan is viable - you can conduct the procedure - that interview scripts, questionnaires, experiments, etc. work appropriately • Iron out problems before doing the main study.
- 34. Controlled Experiments • Designer of a controlled experiment should carefully consider: • proposed hypothesis • selected subjects • measured variables • experimental methods • data collection • data analysis
- 35. Subjects • The choice of subjects is critical to the validity of the results of an experiment • subjects group should represent expected user population expected user population • Consider subject factors such as: • age group, education, skills, culture, technology background • The sample size should be large enough (10+) to be statistically representative of the user population
- 36. Hypothesis and Variables • Hypothesis: prediction of the experiment outcome • Experiments manipulate and measure variables under controlled conditions • There are two types of variables • independent: variables that are manipulated to create different experimental conditions • e.g. number of items in menus, colour of the icons • dependent: variables that are measured to find out the effects of changing the independent variables • e.g. speed of menu selection, speed of locating icons
- 37. Experimental Methods Randomly assigned Statistical data analysis Experimental task Condition 2 Condition 3 Condition 1 Subjects data data data Between- groups Randomly assigned Statistical data analysis Subjects data data data Within- groups Experimental tasks Condition 2 Condition 3 Condition 1 Experimental tasks Condition 1 Condition 3 Condition 2 Experimental tasks Condition 1 Condition 2 Condition 3
- 38. Data Types • Subjective (Qualitative) • Subjective survey • Likert Scale, condition rankings • Observations • Think Aloud • Interview responses • Objective (Quantitative) • Performance measures • Time, accuracy, errors • Process measures • Video/audio analysis How easy was the task 1 2 3 4 5 Not very easy Very easy
- 39. VR APPLICATIONS
- 41. Virtual Reality Applications • Ideal applications for VR should: • Be strongly visual, have 3D spatial elements • Benefit from first person immersion • Benefit from 3D manipulation/navigation • Support Autonomy, Interaction and Presence (AIP Cube) • Etc..
- 42. Not Everything Should be Done in VR Dr Fun - 1990
- 43. Many Possible Types of VR Applications From https://www.slideshare.net/ampnewventures/virtual-reality-vr-continuum-amp-new-ventures
- 45. Potential Disruption for Existing Domains https://www.slideshare.net/BDMIFund/the-emerging-virtual-reality-landscape-a-primer
- 46. Example VR Applications • Education • Google Expeditions • Medicine • Virtual Characters • Entertainment • The Void, Zero Latency • Art + Design • Tilt Brush • Collaboration • Facebook Spaces
- 47. EDUCATION
- 48. Google Expeditions • https://edu.google.com/expeditions/ • Mobile VR Educational application (Android, iOS) • Designed for classroom experiences
- 49. Google Expeditions • Goal: Provide low cost educational VR experience • Based on Google Cardboard VR platform • Different roles: • Guide— person leading an expedition on a tablet • Explorer— person following an expedition on a phone. • Usage • Used by millions of students • Over 1000 educational experiences developed • Royal Collection Trust, American Museum of Natural History, etc.
- 50. Teacher Led VR Experiences • Teacher/Guide uses tablet to control the experience • Selects the virtual tour experience • Guide sees tour script, can select immersive scenes to view • Guide sees focus point and where individual students are looking • Students connect as followers, look at what guides highlight Guide Interface
- 51. System • Hardware • Google Cardboard mobile viewer • Smart phones + tablet (class set) • Wireless router • Software • Viewer and Guide applications (iOS/Android) • 360 image/video VR experiences Class set for 30 students
- 52. Example Experiences • Over 1000 locations/experiences • Great barrier reef, Great Wall of China, Grand Canyon, etc.
- 54. Feedback • Teacher/student survey (100 people) • 65% experienced a “Wow” moment during Google expedition • Noted the variety of educator styles and approaches possible • People enjoyed “The feeling of ‘being’ there” From https://www.slideshare.net/zoesujon/google-expeditions-virtual-reality-and-the-classroom
- 55. Limitations • But 53% of participants identified some problems: • Difficult for some people who wore glasses • Some complained of eye strain, headaches or nausea • Some staff were reluctant/resistant to use the leader tablet • Issues of disabilities and inclusion
- 56. Key Findings • Low-cost VR/mobile VR can provide a valuable educational experience • Visit different locations, different times, etc. • Teach interaction key • Acting as guide, providing educational context • VR requires more work • Address simulator sickness, ergonomic issues, etc. • Immersion/Presence creates learning • Immersion creates memorable educational experience
- 57. Challenges/Solutions • Making VR accessible • Designing for phones, tablets, low cost viewers • Synchronizing content with all viewers • Teacher controlled viewing • Teacher can guide experiences • Engaging interaction on simple viewers • Head pointing based interaction, button input • Supporting Educational goals • Providing compelling educational content
- 58. MEDICINE
- 59. Virtual Patients • Problem • Many doctors have poor doctor/patient skills • Have limited opportunity during training to learn skills • Solution • Virtual patients that doctors can communicate with naturally • Artificial agents with speech understanding
- 60. Typical System Setup • Trainee in front of projection screen • Speech and gesture recognition • Intelligent agent on screen Johnsen, K., Raij, A., Stevens, A., Lind, D. S., & Lok, B. (2007, April). The validity of a virtual human experience for interpersonal skills education. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 1049-1058). ACM.
- 62. Key Findings • Virtual Humans can replace actors in training • interaction skills used with a virtual human translate to the interaction skills used with a real human • Students feel a strong sense of co-presence • Having character respond to speech and gesture increases immersion • VR is capable of creating realistic characters • Life size, intelligent backend, speech recognition • Skills learnt transfer to real world
- 63. Challenges/Solutions • Training in medical environment • Design for training in medical exam room • Use projected VR not HMDs • Natural interaction • Support speech and gesture interaction • Tactile/haptic feedback • Use prosthetics to add support for palpation and other tactile interaction between doctor and virtual patient • Supporting Educational goals • Give virtual character domain knowledge
- 64. ENTERTAINMENT
- 65. Large Scale VR Gaming • Provide multi-player VR gaming in warehouse space • Examples • The Void - https://www.thevoid.com/ • Zero Latency - https://zerolatencyvr.com/
- 66. Typical System • Wide Area Tracking • Computer vision, lights/reflective balls • > 120 cameras for 300 m2 space • Backpack VR system • Haptic feedback, wireless HMD • Real Props • Tracked objects, walls Tracking cameras Backpack system
- 67. The Void Demo • https://www.youtube.com/watch?v=XgetffuOgBA
- 68. Key Findings • Wide area tracking possible • vision based systems can create large scale wide areas tracking, fast enough for game play • Shared gameplay improves experience • Focus on collaborative experiences, using avatar representations and roll division • Haptic feedback significantly increases presence • Use of physical props (objects, walls) • Content is king • Systems need compelling content/game place
- 69. Challenges/Solutions • Wide area tracking • Computer vision tracking of people • Over 100 cameras + multiple servers • Freedom of movement • Custom wireless VR backpacks • Ruggedized HMDs, weapon props • Natural interaction • Redirected walking, tangible props • Compelling content • Multi-sensory feedback, custom game platform
- 70. ART + DESIGN
- 71. Tilt Brush • Intuitive 3D immersive drawing/sculpting program • Developed by Patrick Hackett and Drew Skillman 2014 • Acquired by Google in 2015/Open sourced 2021 • https://www.tiltbrush.com/
- 72. Functionality • Goal: Extremely natural 3D painting/sculpting • User Interface • Two handed interface designed for two controllers (Vive, Rift) • Brush in dominant hand, tool palette in non-dominant • Typical drawing functionality – color, brush width, undo/redo, etc.. • Content sharing • Created content can be exported/shared in 2D/3D formats
- 75. Example Tilt Brush Sketches
- 76. Key Findings • Use familiar tools • Tilt brush interface has familiar sculpting/painting tools – e.g. brush size, colour pallet, etc • Use intuitive interface • Two handed tools with natural metaphor – one hand for pallet/menu, one hand for painting/sculpting • Provide Magical experience • Provide experience not possible in real world, e.g. changing body scale, painting in 3D, etc. • Create a community • Provide ways for people to share content
- 77. Challenges/Solutions • Intuitive Interface • Very natural metaphor – painting in space • Two handed interface – map to VR controllers • Familiar menu objects from paint programs • Need for limited training • Provide in app training, tool tips • Content sharing • Enable content to be exported in variety of formats • Video, animated GIFs, 2D images, 3D files • Engaging Experience • Provides novel immersive artistic experience
- 78. COLLABORATION
- 80. “Only through communication can Human Life hold meaning.” Paulo Freire
- 81. A wide variety of communication cues used. Speech Paralinguistic Para-verbals Prosodics Intonation Audio Gaze Gesture Face Expression Body Position Visual Object Manipulation Writing/Drawing Spatial Relationship Object Presence Environmental Face to Face Communication
- 82. Face to Face Communication Audio Cues Visual Cues Environmental Cues
- 83. Remote Communication Audio Cues Visual Cues Environmental Cues
- 84. Evolution of Communication Tools
- 85. My Workplace in 2020
- 87. Zoom Fatigue..
- 88. Why is Video Conferencing so Tiring? • Physical factors • Poor posture • Staring at fixed screen • Longer hours/work anytime • Cognitive factors • Loss of non-verbal cues, eye gaze • Need to focus more intently • Multi-tasking/split attention • Social factors • Aware of being watched • Seeing yourself • Constant staring “It's almost like you're emoting more because you're just a little box on a screen”. “I’m just so tired.” “You have to fill in the gaps. And that takes cognitive energy. You get tired more quickly.” “For somebody who’s really dependent on those non-verbal cues, it can be a big drain not to have them.”
- 91. Communication Seam • da Task Space Communication Space
- 92. Limitations with Current Technology •Lack of spatial cues • Person blends with background •Poor communication cues • Limited gaze, gesture, non-verbal communication •Introduction of artificial seams • Separation of task/communication space
- 93. Facebook Spaces • Collaborative VR environment • VR meeting and interaction space (up to 4 people) • Focus on communication • Speech and gesture based • https://www.facebook.com/spaces
- 94. System Interaction • Designed for Oculus Rift/HTC Vive • Upper body tracking, touch controllers • Simple interaction • Loading scenes, direct object manipulation • Content creation • Selfie pictures, simple sketching
- 96. Facebook Workrooms • Designed to support meetings • Meeting seating • Shared blackboard • Limited movement • Support for real devices • Keyboard/mouse input • Calibrated computer screen • Private space/public space • Rich communication cues • Lip sync • Natural gestures
- 98. Hybrid Interfaces • Support users on AR/VR or desktop/mobile devices • Spatial.io • AR/VR interface • Screen based interface • Realistic avatars/quick avatar capture • Similar interaction regardless of device • Easy content loading
- 99. Hybrid Interfaces - Spatial https://www.youtube.com/watch?v=PG3tQYlZ6JQ
- 100. Holoportation (2016) • Augmented Reality + 3D capture + high bandwidth • http://research.microsoft.com/en-us/projects/holoportation/
- 103. Other Examples • Many other examples of collaborative VR • Rec Room, High Fidelity, AltspaceVR • Sansar, VR chat, etc..
- 104. Mozilla Hubs • Web based social VR • Customizable avatars, spatial audio, 3D environments • Multi-device support • HMDs, Desktop, Mobile • Open source • https://hubs.mozilla.com
- 106. Mozilla Hubs Spoke Creator • Drag and drop scene editing • Upload assets • 3D objects, textures, animaton • Customize content – Sketchup, Google Poly, Sketchfab, etc
- 108. Mozilla Hubs - Demo
- 109. Key Findings • Minimal social cues okay • Even simple avatars can provide rich social experience • Create shared social context • Important to place users in same shared Virtual Reality environment/shared social context • Audio is key • Provide low latency audio, spatial audio cues • Create a reason for communicating • Why should people want to connect? Create shared activity/reason for people to conference
- 110. Challenges/Solutions • Create shared sense of Presence • Use common background, shared objects • Natural communication • Support non-verbal behaviour, speech/gesture input • Intuitive interaction • Map real body motion onto Avatars • Limited ability to navigate/move through environment • Engaging Experience • Shared content creation, experience capture
- 111. OTHER APPLICATIONS
- 112. Collisions – Australian VR Film • http://www.collisionsvr.com/
- 114. Best VR Apps of 2019 (Digital Trends) • ALLUMETTE – VR Stop motion film • Google Earth – Travel/geography • Kingspray Graffiti – Art/content creation • The FOO Show - VR Talk show • Virtual Desktop – Use desktop in VR • www.digitaltrends.com/virtual-reality/best-virtual-reality-apps/
- 120. RESEARCH DIRECTIONS IN MIXED REALITY Mark Billinghurst [email protected] October 19th 2021
- 121. 1967 – IBM 1401 – half of the computers in the world, $10,000/month to run
- 122. Sketchpad (1963) • Ivan Sutherland • First interactive graphics • Pen based input
- 123. The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal.. Sutherland, Ivan. "The ultimate display." (1965).
- 124. Sutherland Display (1968) https://www.youtube.com/watch?v=NtwZXGprxag
- 125. Star Trek – HoloDeck (1974)
- 126. Star Wars – Hologram (1977)
- 127. My First VR Experience - 1990 • Silicon Graphics Reality Engine • 500,000 polygons/second • VPL Eyephone HMD • 320 x 240 resolution • Magnetic tracking • Glove input • Expensive - $250,000+
- 128. Star Wars – Collaborative AR
- 129. 1999 – Shared Space Demo • Face to face collaborative AR like Star Wars concept
- 130. Shared Space Demo
- 131. Marker Based Tracking: ARToolKit (1999) https://github.com/artoolkit
- 132. CPU: 300 Mhz HDD; 9GB RAM: 512 mb Camera: VGA 30fps Graphics: 500K poly/sec 1998: SGI O2 2008: Nokia N95 CPU: 332 Mhz HDD; 8GB RAM: 128 mb Camera: VGA 30 fps Graphics: 2m poly/sec By 2008 phones had the same hardware as used in Shared Space demo
- 133. 2005: Mobile AR version of Shared Space • AR Tennis • Shared AR content • Two user game • Audio + haptic feedback • Bluetooth networking
- 134. ARTennis Demo
- 135. VR and AR Today • Large growing market • > $25 Billion USD in 2020 • Hundreds of millions of users • Many available devices • HMD, phones, tablets, HUDs • Robust developer tools • Vuforia, MRTK, Unity, etc • Large number of applications • > 150K developers, > 100K apps • Strong research/business communities • ISMAR, IEEE VR, AWE conferences, AugmentedReality.org, etc
- 139. Future Visions of VR: Ready Player One • https://www.youtube.com/watch?v=LiK2fhOY0nE
- 140. Today vs. Tomorrow VR in 2021 VR in 2045 Graphics High quality Photo-realistic Display 110-150 degrees Total immersion Interaction Handheld controller/some gesture Full gesture/body/gaze Navigation Limited movement Natural Multiuser Few users Millions of users
- 143. “.. the technologies that will significantly affect our lives over the next 10 years have been around for a decade. The future is with us ... The trick is learning how to spot it” October 2004 Bill Buxton
- 144. CONCLUSIONS
- 145. Conclusions • AR/VR/MR is becoming commonly available • Significant advances over 50+ years • In order to achieve Sutherland’s vision, need research in • Display, Tracking, Input • New MR technologies will enable this to happen • Display devices, Interaction, Tracking technologies • There are still significant areas for research • Social Acceptance, Perception, Collaboration, Etc.
- 146. More Information Billinghurst, M. (2021). Grand Challenges for Augmented Reality. Frontiers in Virtual Reality, 2, 12.