ar-mr(1) - 3D User Interfaces for the Real 3D World Lecture...

Info iconThis preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: 3D User Interfaces for the Real 3D World Lecture #16: Augmented/Mixed Reality Spring 2009 Joseph J. LaViola Jr. Special thanks to Ivan Poupyrev Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Definitions Augmented reality: Refers to a system Augmented in which the user views and acts within an enhanced version of the real world. The enhanced enhancements are virtual (computergenerated), and can include objects or information. Mixed reality: Refers to a system that Mixed combines real and virtual objects and information. Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 1 Mixed Reality Continuum Mixed Reality (MR) Reality Augmented Reality (AR) Augmented Virtuality (AV) Virtuality Milgram (1994) Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. AR/MR Application Areas Maintenance Maintenance Training Training Tourism / Cultural heritage Tourism Design / construction Design Battlefield information display Battlefield Entertainment Entertainment Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 2 AR/MR Technology - Displays See-through HMDs: See Video see-through Video seeOptical see-through Optical see- Handheld displays Handheld Projection Projection Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. AR/MR Technology - Tracking Optical / vision-based Optical tracking AR toolkit AR ensures portability ensures large number of large tracked objects Registration and low Registration latency are crucial for crucial AR systems Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 3 AR/MR technology - Tracking Sourceless inertial orientation tracking Sourceless GPS position tracking GPS enables mobile outdoor AR enables Markerless tracking Markerless Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Mobile outdoor AR “Backpack systems” User wears/carries: User Computer Computer HMD HMD Inertial tracker Inertial GPS unit/antenna GPS Input device(s) Input Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 4 Mixed Reality Interfaces Azuma (1997) Azuma combine real and combine virtual objects interactive in real time interactive virtual objects are virtual registered in 3D physical world KARMA, Feiner, et al. 1993 Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Challenges in AR Interfaces Conflict between real Conflict world and virtual not neatly separated not anymore Limitations of displays Limitations precise, fast registration & precise, tracking spatially seamless display spatially Limitations of controllers Limitations precise, fast registration & precise, tracking spatially seamless interactivity spatially Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality Image Copyright Sony CSL ©Joseph J. LaViola Jr. 5 AR Interfaces as 3D Information Browsers (I) 3D virtual objects are 3D registered in 3D see-through HMDs, 6DOF seeHMDs, optical, magnetic trackers “VR in Real World” Interaction Interaction 3D virtual viewpoint 3D control Applications Applications visualization, guidance, visualization, training Spring 2009 State, et al. 1996 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. AR Interfaces as Context-Based Information Browsers (II) Information is registered to realInformation realworld context Hand held AR displays Hand Video-see-through (Rekimoto, Video- see(Rekimoto, 1997) or non-see through non(Fitzmaurice, et al. 1993) magnetic trackers or computer magnetic vision based Interaction Interaction manipulation of a window manipulation into information space Applications Applications context-aware information displays context- Spring 2009 Rekimoto, et al. 1997 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 6 AR Info Browsers (III): Pros and Cons Important class of AR Important interfaces wearable computers wearable AR simulation, training AR Limited interactivity Limited modification and modification authoring virtual content is difficult Spring 2009 Rekimoto, et al. 1997 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 3D AR Interfaces (I) Virtual objects are displayed in 3D Virtual space and can be also manipulated in 3D see-through HMDs and 6DOF seehead-tracking for AR display head6DOF magnetic, ultrasonic, or 6DOF other hand trackers for input Interaction Interaction viewpoint control viewpoint 3D user interface interaction: 3D manipulation, selection, etc. Spring 2009 Kiyokawa, et al. 2000 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 7 3D AR Interfaces (II): Information Displays How to move information in AR How context dependent information browsers? InfoPoint (1999) InfoPoint hand-held device hand- computer-vision 3D computertracking moves augmented data moves between marked locations HMD is not generally HMD needed, but desired since there are little display capabilities Spring 2009 Khotake, et al. 1999 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 3D AR Interfaces (III): Pros and Cons Important class of AR interfaces Important entertainment, design, training entertainment, Advantages Advantages seamless spatial interaction: User can interact with 3D seamless virtual object everywhere in physical space natural, familiar interfaces natural, Disadvantages Disadvantages usually no tactile feedback and HMDs are often required usually interaction gap: user has to use different devices for interaction virtual and physical objects Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 8 Tangible interfaces and augmented surfaces (I) Basic principles Basic virtual objects are projected virtual on a surface back projection back overhead projection overhead physical objects are used as physical controls for virtual objects tracked on the surface tracked virtual objects are registered to virtual the physical objects physical embodiment of the user physical interface elements collaborative collaborative Digital Desk. 1993 Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Tangible Interfaces and Augmented Surfaces (II) Graspable interfaces, Bricks system Graspable (Fitzmaurice, et al. 1995) and Tangible interfaces, e.g. MetaDesk (Ullmer’97): back-projection, infrared-illumination computer backinfraredvision tracking physical semantics, tangible handles for virtual physical interface elements metaDesk. 1997 Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 9 Tangible Interfaces and Augmented Surfaces (III) Rekimoto, et al. Rekimoto 1998 front projection front marker-based tracking markermultiple projection surfaces multiple tangible, physical tangible, interfaces + AR interaction with computing devices Augmented surfaces, 1998 Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Tangible Interfaces and Augmented Surfaces (IV) Advantages Advantages seamless interaction flow – user hands are seamless used for interacting with both virtual and physical objects. no need for special purpose input devices no Disadvantages Disadvantages interaction is limited only to 2D surface interaction spatial gap in interaction - full 3D interaction spatial and manipulation is difficult Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 10 Orthogonal Nature of AR Interfaces (Poupyrev, 2001) 3D AR No interaction is everywhere Spatial gap Interaction gap Spring 2009 Augmented surfaces Yes interaction is only on 2D surfaces Yes separate devices for physical and virtual objects No same devices for physical and virtual objects CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Tangible AR interfaces (I) Virtual objects are registered to marked Virtual physical “containers” HMD HMD video-see-through tracking and registration video- seeusing computer vision tracking Virtual interaction by using Virtual 3D physical container tangible, physical interaction tangible, 3D spatial interaction 3D Collaborative Collaborative Shared Space, 1999 Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 11 Tangible AR (II): Generic Interface Semantics Tiles semantics Tiles data tiles data operation tiles operation menu menu clipboard clipboard trashcan trashcan help help Operation on tiles Operation proximity proximity spatial arrangements spatial space-multiplexed space- Spring 2009 Tiles, 2001 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Tangible AR (III): Space-Multiplexed Data authoring in Tiles (Poupyrev, et al. 2001). Left, outside view of the system; right, view of the left participant. Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 12 Tangible AR (IV): TimeMultiplexed Interaction Data authoring in WOMAR interfaces (Kato et al. 2000). The user can pick, manipulate and arrange virtual furniture using a physical paddle. Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Tangible AR (V): AR - VR Transitory Interfaces Magic Book (Billinghurst, Magic et al. 2001) 3D pop-up book: a 3D poptransitory interfaces augmented Reality augmented interface portal to Virtual portal Reality immersive virtual immersive reality experience collaborative collaborative Spring 2009 Augmented Reality Virtual Reality CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 13 Tangible AR (VI): Conclusions Advantages Advantages seamless interaction with both virtual and seamless physical tools no need for special purpose input devices no seamless spatial interaction with virtual objects seamless 3D presentation of and manipulation with virtual 3D objects anywhere in physical space Disadvantages Disadvantages required HMD required markers should be visible for reliable tracking markers Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Interfaces for Mobile Outdoor AR Devices must be Devices handheld No tracking or limited No tracking for devices Interaction at-aInteraction distance Tinmith project Tinmith Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 14 Challenges in AR/MR Occlusion and depth perception Occlusion Text display and legibility Text Visual differences between real and virtual Visual objects Registration and tracking Registration Bulky HMDs and other equipment Bulky Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. Next Class Paper presentations begins Paper Final project proposals due Friday (3-21-08)!!! Final Spring 2009 CAP6938 – 3D User Interfaces for Games and Virtual Reality ©Joseph J. LaViola Jr. 15 ...
View Full Document

Ask a homework question - tutors are online