Tangible AR Interface
- 1. Augmented Reality Interfaces for Visual Analytics Mark Billinghurst HIT Lab NZ University of Canterbury
- 4. We are here
- 7. What Year is This From ? 1982 2.77 Mhz
- 10. Invisible Interfaces Jun Rekimoto, Sony CSL
- 11. Augmented Reality • Augmented Reality: Enhances Reality • Key Characteristics • Combines real and virtual images • Interactive in Real-Time • Registered in 3D
- 12. Medical Imaging Devices X-ray, US, CT, PET, MR, PET/CT, (Capsule) Endoscopy, Optical Imaging, Open MR, 4D US, …
- 13. Medical Imaging Data X-ray, Contract Agents, Image intensifier, US, CT, PET, MR, …
- 14. 3D imaging data is often displayed in 2D!
- 15. There are as many displays as imaging modalities!
- 16. Current CAS interfaces MediVisi on
- 17. AR Medical Trials Sauer et al. 2000 at Siemens Corporate Research, NJ Stereo video see through F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial, MICCAI 2002
- 18. AR History Summary 1960’s – 80’s: Early Experimentation 1980’s – 90’s: Basic Research Tracking, displays 1995 – 2005: Tools/Applications Interaction, usability, theory 2005 - : Commercial Applications Games, Medical, Industry
- 19. MIT Technology Review March 2007 list of the 10 most exciting technologies
- 20. Visual Analytics Prototype using Tangible AR Prototype developed by Nick Hedley, at SFU. Design based on empirical findings of several tangible AR studies compared with other conventional 3D visualizations Image courtesy of N. Hedley / SIRL (2008)
- 22. Interface Design Path 1/ Prototype Demonstration 2/ Adoption of Interaction Techniques from other interface metaphors Augmented Reality 3/ Development of new interface metaphors appropriate to the medium Virtual Reality 4/ Development of formal theoretical models for predicting and modeling user actions Desktop WIMP
- 23. AR Design Principles Interface Components Physical components Display elements - Visual/audio Interaction metaphors Physical Display Elements Interaction Elements Metaphor Input Output
- 24. AR User Interface Design Space Reality Virtual Reality Augmented Reality Physical Design Virtual Design NO STANDARD PRINCIPLES !! STANDARD PRINCIPLES STANDARD PRINCIPLES SOME CLUES
- 25. Tangible Interfaces (Ishii 97) Create digital shadows for physical objects Foreground graspable UI Background ambient interfaces
- 26. Tangible Interfaces Dangling String Jeremijenko 1995 Ambient ethernet monitor Relies on peripheral cues Ambient Fixtures Dahley, Wisneski, Ishii 1998 Use natural material qualities for information display
- 27. TerraVision (ART+COM 1994) Goal use virtual earth to visualize information Features seamless zooming tangible interface separation of physical + virtual
- 28. Lessons from Tangible Interfaces Physical objects make us smart Norman’s “Things that Make Us Smart” encode affordances, constraints Objects aid collaboration establish shared meaning Objects increase understanding serve as cognitive artifacts
- 29. Limitations Difficult to change object properties can’t tell state of digital data Limited display capabilities pinwheels = 1D, projection screen = 2D dependent on physical display surface Separation between object and display TerraVision
- 30. Back to the Real World AR overcomes limitation of TUIs enhance display possibilities merge task/display space provide public and private views TUI + AR = Tangible AR Apply TUI methods to AR interface design
- 31. Tangible AR Design Principles Tangible AR Interfaces use TUI principles Physical controllers for moving virtual content Support for spatial 3D interaction techniques Time and space multiplexed interaction Support for multi-handed interaction Match object affordances to task requirements Support parallel activity with multiple objects Allow collaboration between multiple users
- 32. Design of Objects Objects Purposely built – affordances “Found” – repurposed Existing – already at use in marketplace Make affordances obvious (Norman) Object affordances visible Give feedback Provide constraints Use natural mapping Use good cognitive model
- 33. Tangible AR: VOMAR Interface (Kato 2000) Use of natural physical object manipulations to control virtual objects VOMAR Demo Catalog book: - Turn over the page Paddle operation: - Push, shake, incline, hit, scoop
- 34. VOMAR Interface
- 35. Case Study: 3D AR Lens Goal: Develop a lens based AR interface MagicLenses Developed at Xerox PARC in 1993 View a region of the workspace differently to the rest Overlap MagicLenses to create composite effects
- 36. 3D MagicLenses MagicLenses extended to 3D (Veiga et. al. 96) Volumetric and flat lenses
- 37. AR Lens Design Principles Physical Components Lens handle - Virtual lens attached to real object Display Elements Lens view - Reveal layers in dataset Interaction Metaphor Physically holding lens
- 38. 3D AR Lenses: Model Viewer Displays models made up of multiple parts Each part can be shown or hidden through the lens Allows the user to peer inside the model Maintains focus + context
- 39. Examples Simple Wireframe Terrain layers Non-photorealistic rendering
- 40. Flexible Lens Red Planet, 2000 (5.2/10 IMDB) Flexible Lens Surface Bimanual interaction Digital paper analogy Looks expensive
- 41. Flexible Lens
- 42. Flexible Lens – Gesture Input Default Stretch Twist Bend Rotate
- 43. Milgram’s Continuum (1994) Mixed Reality (MR) Reality Augmented Augmented Virtuality (Tangible Reality (AR) Virtuality (AV) (Virtual Interfaces) Reality)
- 45. Collaboration
- 46. Communication Cues A wide variety of communication cues used. Visual Audio Gaze Speech Gesture Paralinguistic Face Expression Paraverbals Body Position Prosodics Intonation Object Manipulation Environmental Writing/Drawing Spatial Relationship Object Presence
- 47. Collaboration in the Future ? Remote Conferencing Face to face Conferencing
- 48. Remote AR Conferencing Moves conferencing from the desktop to the workspace
- 49. Face-to-face collaboration People surround a table It is easy to see each other Communication Space Task Space Computer supported collaboration People sit side by side It is hard to see each other Communication Space Task Space
- 50. Collaborative Augmented Reality Seamless Interaction Natural Communication Attributes Virtuality Augmentation Cooperation Independence Individuality
- 51. Hybrid User Interfaces 1 2 3 4 PERSONAL TABLETOP WHITEBOARD MULTIGROUP Private Display Private Display Private Display Private Display Group Display Public Display Group Display Public Display
- 53. Shared Design Space (M. Haller, Hagenberg) Enhanced face to face collaboration Tangible tools Digital pens Active desktop Multiple users
- 54. Digital Pen, ANOTO Captures position (x, y) in absolute coordinates, time (t), pressure (p), and status (up, down) www.anoto.com
- 58. Presentation Area Design Area Shared Workspace Private Workspace
- 59. Evaluation
- 60. Interaction Design is All About You Users should be involved throughout the Design Process Consider all the needs of the user
- 64. Survey of AR Papers Edward Swan (2005) Surveyed major conference/journals (1992-2004) - Presence, ISMAR, ISWC, IEEE VR Summary 1104 total papers 266 AR papers 38 AR HCI papers (Interaction) 21 AR user studies Only 21 from 266 AR papers have formal user study (<8% of all AR papers)
- 65. AR Papers
- 66. Types of Experiments Perception How is virtual content perceived ? What perceptual cues are most important ? Interaction How can users interact with virtual content ? Which interaction techniques are most efficient ? Collaboration How is collaboration in AR interface different ? Which collaborative cues can be conveyed best ?
- 67. Looking to the Future
- 68. The Future is with us It takes at least 20 years for new technologies to go from the lab to the lounge.. “The technologies that will significantly affect our lives over the next 10 years have been around for a decade. The future is with us. The trick is learning how to spot it. The commercialization of research, in Oct 11th 2004 other words, is far more about prospecting than alchemy.” Bill Buxton
- 69. VR vs. AR Don’t oversell it.. Be Honest
- 70. Mobile Phone AR Mobile Phones camera processor display AR on Mobile Phones Simple graphics Optimized computer vision Collaborative Interaction
- 72. Collaborative AR AR Tennis Virtual tennis court Two user game Audio + haptic feedback Bluetooth messaging
- 73. Collaborative AR
- 74. UbiVR Cellphone for service discovery Visualization of real world sensor data
- 75. Real Time Rome (Ratti, MIT)
- 76. System Design
- 77. Results
- 79. NZ Cell Tower Locations
- 83. Physical + Social Networking Location based social networking Where are my friends now? Where are people with common interests? Trend analysis How do these people behave? Where is my friend likely to be?
- 84. Conclusions
- 85. Conclusions AR allows us to return to the real world Invisible Interfaces Enhanced Interaction Enhanced Collaboration Future Research Directions Evaluation Massive AR
- 86. More Information • Mark Billinghurst – [email protected] • Websites – www.hitlabnz.org