Interaction 3D Techniques for Virtual Environments: Selection and Manipulation
Doug A. Bowman
1
Terminology
Interaction Technique (IT) method for accomplishing a task 3D application system that displays 3D information 3D interaction performing user actions in three dimensions
(C) 2006 Doug Bowman, Virginia Tech
2
2
Didnt we already cover input devices?
User interface software ITs Input devices
System Software
Output devices
User
(C) 2006 Doug Bowman, Virginia Tech
3
3
Video example: Virtual-SAP
(C) 2006 Doug Bowman, Virginia Tech
4
4
Universal interaction tasks
Navigation
Travel motor component Wayfinding cognitive component
Selection Manipulation System control Symbolic input
(C) 2006 Doug Bowman, Virginia Tech 5
Well be considering four basic or universal interaction tasks for 3D/VE applications. Obviously, there are other tasks which are specific to an application domain, but these are some basic building blocks that can often be combined to create a more complex task. Navigation is the most common VE task, and is actually composed of two tasks. Travel is the motor component of navigation, and just refers to the physical movement from place to place. Wayfinding is the cognitive or decision-making component of navigation, and it asks the questions, where am I?, where do I want to go?, how do I get there?, and so on. Selection is simply the specification of an object or a set of objects for some purpose. Manipulation refers to the specification of object properties (most often position and orientation, but also other attributes). Selection and manipulation are often used together, but selection may be a stand-alone task. For example, the user may select an object in order to apply a command such as delete to that object. System control is the task of changing the system state or the mode of interaction. This is usually done with some type of command to the system (either explicit or implicit). Examples in 2D systems include menus and command-line interfaces. It is often the case that a system control technique is composed of the other three tasks (e.g. a menu command involves selection), but its also useful to consider it separately since special techniques have been developed for it and it is
5
Selection & Manipulation
Selection: specifying one or more objects from a set Manipulation: modifying object properties
(position, orientation, scale, shape, color, texture, behavior, etc.)
(C) 2006 Doug Bowman, Virginia Tech
6
6
Goals of selection
Indicate action on object Query object Make object active Travel to object location Set up manipulation
(C) 2006 Doug Bowman, Virginia Tech
7
7
Selection performance
Variables affecting user performance
Object distance from user Object size Density of objects in area Occluders
(C) 2006 Doug Bowman, Virginia Tech
8
8
Common selection techniques
Touching with virtual hand Ray/cone casting Occlusion / framing Naming Indirect selection
(C) 2006 Doug Bowman, Virginia Tech
9
9
Enhancements to basic techniques
Arm-extension
Mapping Reeling
2D / 3D World in Miniature
Select iconic objects
(C) 2006 Doug Bowman, Virginia Tech
10
10
Ray-casting and Go-Go video
(C) 2006 Doug Bowman, Virginia Tech
11
11
Go-Go implementation
Requires torso position t - tracked or inferred Each frame:
Get physical hand position h in world CS Calculate physical distance from torso dp = dist(h, t) Calculate virtual hand distance dv = gogo(dp) h"t Normalize torso-hand vector th = h"t V. hand position v = t + dv *(h-t) (in world CS)
!
(C) 2006 Doug Bowman, Virginia Tech 12
To implement Go-Go, we first need the concept of the position of the users body. This is needed because we stretch our hands out from the center of our body, not from our head (which is usually the position that is tracked). I have implemented this using an inferred torso position, which is defined as a constant offset in the negative y direction from the head. You could also place a tracker on the users torso. Before rendering each frame, you get the physical hand position in the world CS, and then calculate its distance from the torso object using the distance formula. The virtual hand distance can then be obtained by applying the function shown in the graph on the previous slide. I have used the function d2.3 (starting at the point (D,D)) as a useful function in my environments, but the exponent used depends on the size of the environment and the desired accuracy of selection at a distance. Now that we know the distance at which to place the virtual hand, we need to determine its position. The most common implementation is to keep the virtual hand on the ray extending from the torso and going through the physical hand. Therefore, if we get a vector between these two points, normalize it, multiply it by the distance, then add this vector to the torso point, we obtain the position of the virtual hand.
12
WIM video
(C) 2006 Doug Bowman, Virginia Tech
13
13
Selection classification
Feedback
graphical tactile audio object touching pointing indirect selection button gesture voice
Selection
Object indication Indication to select
(C) 2006 Doug Bowman, Virginia Tech
14
14
Evaluation: Selection Task
Ray-casting and image-plane generally more effective than Go-Go Exception: selection of very small objects can be more difficult with pointing Ray-casting and image-plane techniques result in the same performance (2DOF) Image-plane technique less comfortable
(C) 2006 Doug Bowman, Virginia Tech
15
This slide summarizes some of the results from the experiments conducted by Poupyrev et al. (1998) and Bowman et al. (1999). Two interaction techniques have been evaluated in experiments by Poupyrev: ray-casting and Go-Go interaction technique with 5 levels of the distance and 3 levels of the object size; within subject design with 12 participants was used in the experiments. Three interaction techniques have been evaluated by Bowman: ray-casting, image plane, and Go-Go techniques with 3 levels of the distance and 2 levels of the object sizes. Between subject design with 5 participants for each technique was used in the experiments Several general conclusions emerge from the experiments (also, see figures below) First, in general purpose selection task pointing techniques demonstrate better performance then Go-Go requiring less physical movement from the user. The only exceptions reported by Poupyrev is when very high precision of selection was required, e.g. for small objects (about 4 degrees of field of view) especially located far away at the distance. In this case pointing was not as effective as Go-Go which is quite intuitive: pointing on small objects is more difficult then on large especially when they are further away. This replicates observations of other researchers (Liang, 1994; Forsberg, 1996); the Fitts law provides a theoretical explanation for this phenomena (in Bowmans experiments the size and the distance did not significantly affected the user performance for pointing techniques). There was significant differences in user performance in these two experiments which can be explained by different implementations of the interaction techniques. For example, in Poupyrevs experiments the ray-casting was implemented as a short ray using which the user pointed at objects. Bowman, on the other hand, implemented raycasting so that the user could actually touch object with a ray, which might be a more effective implementation of this technique. Furthermore, different settings in the Go-Go technique can also greatly affect user performance.
15
Implementation issues for selection techniques
How to indicate selection event Object intersections Feedback
Graphical Aural Tactile
Virtual hand avatar List of selectable objects
(C) 2006 Doug Bowman, Virginia Tech 16
There are several common issues for the implementation of selection techniques. One of the most basic is how to indicate that the selection event should take place (e.g. you are touching the desired object, now you want to pick it up). This is usually done via a button press, gesture, or voice command, but it might also be done automatically if the system can infer the users intent. You also have to have efficient algorithms for object intersections for many of these techniques. Well discuss a couple of possibilities. The feedback you give to the user regarding which object is about to be selected is also very important. Many of the techniques require an avatar (virtual representation) for the users hand. Finally, you should consider keeping a list of objects that are selectable, so that your techniques do not have to test every object in the world for selection, increasing efficiency.
16
Goals of manipulation
Object placement
Design Layout Grouping
Tool usage Travel
(C) 2006 Doug Bowman, Virginia Tech
17
17
Manipulation metaphors
Simple virtual hand
Natural but limited
Ray casting
little effort required Exact positioning and orienting very difficult (lever arm effect)
(C) 2006 Doug Bowman, Virginia Tech
18
18
Manipulation metaphors II
Hand position mapping
Natural, easy placement Limited reach, fatiguing, overshoot
Indirect depth mapping
Infinite reach, not tiring Not natural, separates DOFs
(C) 2006 Doug Bowman, Virginia Tech
19
19
HOMER technique
Hand-Centered Object Manipulation Extending Ray-Casting Select: ray-casting Manipulate: hand
Time
(C) 2006 Doug Bowman, Virginia Tech
20
20
Manipulation metaphors III
HOMER (ray-casting + arm-extension)
Easy selection & manipulation Expressive over range of Hard distances to move objects away from you
(C) 2006 Doug Bowman, Virginia Tech
21
21
HOMER video
(C) 2006 Doug Bowman, Virginia Tech
22
22
HOMER implementation
Requires torso position t Upon selection, detach virtual hand from tracker, move v. hand to object position in world CS, and attach object to v. hand (w/out moving object) Get physical hand position h and distance dh = dist(h, t) Get object position o and distance do = dist(o, t)
(C) 2006 Doug Bowman, Virginia Tech 23
Like Go-Go, HOMER requires a torso position, because you want to keep the virtual hand on the ray between the users body (torso) and the physical hand. The problem here is that HOMER moves the virtual hand from the physical hand position to the object upon selection, and it is not guaranteed that the torso, physical hand, and object will all line up at this time. Therefore, in my implementation, I calculate where the virtual hand would be if it were on this ray initially, then calculate the offset to the position of the virtual object, and maintain this offset throughout manipulation. When an object is selected via ray-casting, you must first detach the virtual hand from the hand tracker. This is due to the fact that if it remained attached but you move the virtual hand model away from the physical hand location, a rotation of the physical hand will cause a rotation and translation of the virtual hand. You then move the virtual hand in the world CS to the position of the selected object, and attach the object to the virtual hand in the scene graph (again, without moving the object in the world CS). To implement the linear depth mapping, we need to know the initial distance between the torso and the physical hand, and between the torso and the selected object. The ratio do/dh will be the scaling factor. 23
HOMER implementation (cont.)
Each frame:
Copy hand tracker matrix to v. hand matrix (to set orientation) Get physical hand position hcurr and distance: dh-curr = dist(hcurr, t) $ do ' V. hand distance d = d ) vh h"curr # & % dh ( h "t thcurr = curr Normalize torso-hand vector hcurr " t V. hand position vh = t + dvh*(thcurr) !
(C) 2006 Doug Bowman, Virginia Tech 24
!
Now, each frame you need to set the position and orientation of the virtual hand. The selected object is attached to the virtual hand, so it will follow along. Setting the orientation is relatively easy. You can simply copy the transformation matrix for the hand tracker to the virtual hand, so that their orientation matches. To set the position, we need to know the correct depth and the correct direction. The depth is found by applying the linear mapping to the current physical hand depth. The physical hand distance is simply the distance between it and the torso, and we multiply this by the scale factor do/dh to get the virtual hand distance. We then obtain a normalized vector between the physical hand and the torso, multiply this vector by the v. hand distance, and add the result to the torso position to obtain the virtual hand position.
24
Manipulation metaphors IV
Scaled-world grab
Easy, natural manipulation User discomfort with use
World-in-miniature
All manipulation in reach Doesnt scale well, indirect
(C) 2006 Doug Bowman, Virginia Tech
25
25
Scaled-world grab video
(C) 2006 Doug Bowman, Virginia Tech
26
26
Technique Classification by metaphor
VE manipulation techniques Exocentric metaphor
World-In-Miniature Scaled-world grab
Egocentric metaphor
Virtual Hand metaphor
"Classical" virtual hand Go-Go Indirect, stretch Go-Go
Virtual Pointer metaphor
Ray-casting Aperture Flashlight Image plane
(C) 2006 Doug Bowman, Virginia Tech 27
This slide presents a simple taxonomy of current VE manipulation techniques. They are categorized according to their basic interaction metaphors into exocentric or egocentric techniques. These two terms originated in the studies of cockpit displays , and are now used to distinguish between two fundamental styles of interaction within VEs. In exocentric interaction, also known as the Gods eye viewpoint, users interact with VEs from the outside (the outside-in world referenced display ); examples are the World-In-Miniature and world scaling techniques. In egocentric interaction, which is the most common in immersive VEs, the user interacts from inside the environment, i.e., the VE embeds the user . There are currently two basic metaphors for egocentric manipulation: virtual hand and virtual pointer. With the techniques based on the virtual hand metaphor, users can reach and grab objects by touching and picking them with a virtual hand. The choice of the mapping function discriminates techniques based on the virtual hand metaphor. With techniques based on the virtual pointer metaphor, the user interacts with objects by pointing at them. The selection volume (i.e. conic or ray), definition of ray direction, and disambiguation function are some of the design parameters for such techniques.
27
Technique Classification by components
Object Attachment
attach to hand attach to gaze hand moves to object object moves to hand user/object scaling no control 1-to-N hand to object motion maintain body-hand relation other hand mappings indirect control no control 1-to-N hand to object rotation other hand mappings indirect control graphical force/tactile audio
Object Position
Manipulation
Object Orientation Feedback
(C) 2006 Doug Bowman, Virginia Tech
28
Component-based (Bowman, 1999) taxonomy is another way to classify the manipulation techniques. The taxonomy is based on the observation that all techniques consist of several basic components with similar purposes. For example, in a manipulation task, the object can be positioned, rotated, and the feedback should be provided, etc. Then, instead of classifying the techniques as a whole, we classify only components that allow to accomplish these subtasks within techniques. Then, theoretically, any technique can be constructed out of these components simply by picking appropriate components and putting them together.
28
Evaluation: positioning task
Ray casting effective if the object is repositioned at constant distance Scaling techniques (HOMER, scaled world grab) difficult in outward positioning of objects: e.g. pick an object located within reach and move it far away If outward positioning is not needed then scaling techniques might be effective
(C) 2006 Doug Bowman, Virginia Tech 29
Evaluating techniques for positioning task is difficult because of the vast amount of variables affecting user performance during the manipulation task: depending of the direction of positioning (e.g. inward or outward, left to right, on the constant distance from the user or with change of distance, and etc.), travel distance, required accuracy of manipulation as well as degrees of freedom, the performance of the user can differ drastically. One of the reasons behind these differences in user performance is different groups of muscles that are come into action for different task conditions. However, there were exploratory evaluations (Poupyrev, et al. 1998, Bowman et al. 1999). These studies found, for example, that that raycasting technique, while not very efficient for positioning with a change of the distance from the user, can be very efficient when the starting and target positions of the object are located at the constant distance. The scaling techniques, such as HOMER (Bowman et al. 1997) and scaled-world grab (Mine, 1998) can not be used to perform outward positioning of objects: i.e. if the user wants to pick an object that is already located within the reach and move it away at-a-distance the coefficient of scaling would be equal to 1 and this operation might be difficult to accomplish. However, if the outward movement are rare and do not occur often then the scaling techniques can be effective.
29
Evaluation: orientation task
Setting precise orientation can be very difficult Shape of objects is important Orienting at-a-distance harder than positioning at-a-distance Techniques should be hand-centered
(C) 2006 Doug Bowman, Virginia Tech
30
30
Manipulation notes
No universally best technique Constraints and reduced DOFs Naturalism not always desirable If VE is not based in the real, design it so manipulation is optimized
(C) 2006 Doug Bowman, Virginia Tech
31
31
Manipulation enhancements
Constraints 2-handed manipulation Haptic feedback Multi-modal manipulation
(C) 2006 Doug Bowman, Virginia Tech
32
32
Implementation issues for manipulation techniques
Integration with selection technique Disable selection and selection feedback while manipulating What happens upon release?
(C) 2006 Doug Bowman, Virginia Tech
33
One important issue for any manipulation technique is how well it integrates with the chosen selection technique. Many techniques, as we have said, do both: e.g. simple virtual hand, ray-casting, and go-go. Another issue is that when an object is being manipulated, you should take care to disable the selection technique and the feedback you give the user for selection. If this is not done, then serious problems can occur if, for example, the user tries to release the currently selected object but the system also interprets this as trying to select a new object. Finally, you should think in general about what happens when the object is released. Does it remain at its last position, possibly floating in space? Does it snap to a grid? Does it fall via gravity until it contacts something solid? The application requirements will determine this choice.
33