Interaction techniques in Virtual Environments
1998
R. van de Pol
Department Computer Graphics & CAD/CAM
Faculty of Technical Mathematics and Informatics
Delft University of Technology
Graphics, Visualization and Usability Center
Georgia Institute of Technology
Preface
This report has been written as a research assignment, which is a required part of the study Technical Informatics of the Delft University of Technology. It is a literature study on interaction techniques in Virtual Environments (VEs) that serves as a basis for my Master’s Thesis project. I will do this project at the Graphics, Visualization and Usability (GVU) Center of the Georgia Institute of Technology. The project involves the research of interaction techniques suitable for the Virtual Workbench, a relatively new interface paradigm in the field of Virtual Environments. The Virtual Workbench will primarily be used for the development applications for fast, exploratory visualization of large data, including terrain data and atmospherical data.
My advisors for this project are ir. F.H. Post (TU Delft) and W. Ribarksy, Ph.D. (GVU Center).
R. van de Pol
March 1998
TU Delft i
Summary
For virtual environments to be effective, users should be able to perform useful tasks in them. This implies that the user should be able to interact with the environment and the objects within it. The ability of the user to maintain a sense of ‘presence’ in the environment is often seen to be the main goal of virtual environment systems. One way to achieve this is to try to mimic real-world interaction tasks in virtual environments by allowing a person to use natural and intuitive gestures and movements. However, this is often too restricting. Therefore, in order to improve task performance, it is useful to extend interaction tasks in virtual environments and enhance them in ways that are not possible in the real world.
A lot of problems exist when interacting with virtual objects and environments. The most important ones are the lack of constraints (such as solid objects and gravity), and technological limitations, which tend to make working in three dimensions very hard. It is important that a unifying framework is developed for interacting in virtual environments, comparable to the WIMP (Windows, Icons, Menus, Pointer) interface in traditional workstation environments. However, because of the very different demands applications can pose on the tasks that should be available to the user, this is very difficult.
This report discusses various interaction techniques that have been developed to perform three basic interaction tasks: navigation, selection, and manipulation. Furthermore, the use of (virtual) menus in virtual environments is investigated. Menus have greatly proven their use in traditional computing environments, and therefore it would seem worthwhile to see if and how they can be useful in virtual environments.
TU Delft ii
Table of Contents
Preface i
Summary ii
1 Introduction 1
1.1 What are virtual environments? 1
1.1.1 Types of VE systems 2
1.1.2 General tasks in VE systems 3
1.2 Hardware devices for VE systems 4
1.2.1 Output devices 4
1.2.2 Input devices 6
1.3 Why is interaction important? 7
1.4 CHIMP and the Virtual Gorilla Exhibit 8
1.4.1 CHIMP 8
1.4.2 Virtual Gorilla Exhibit 9
2 Interaction in VEs 11
2.1 Presence 11
2.2 Interaction in VEs: why is it hard? 13
2.3 Requirements on interaction in VEs 13
2.4 Basic interaction tasks 16
3 Navigation 17
3.1 Issues of navigation 17
3.1.1 What is navigation? 17
3.1.2 Situation awareness 18
3.1.3 Constraints on navigation 19
3.2 Specifying direction 19
3.2.1 Physical movement 20
3.2.2 Hand directed techniques 21
3.2.3 Gaze directed techniques 22
3.2.4 Object driven techniques 24
3.2.5 Goal driven techniques 25
3.2.6 Dynamic scaling 25
3.2.7 World in miniature (WIM) 25
3.3 Specifying speed 27
3.3.2 Predictor based technique 28
3.4 Navigation in CHIMP and the Virtual Gorilla Exhibit 29
3.4.1 CHIMP 29
3.4.2 Virtual Gorilla Exhibit 30
4 Selection 32
4.1 Issues of selection 32
4.2 Selection techniques 33
4.2.1 Physical intersection 33
4.2.2 Scaled world grab 34
4.2.3 Silk cursor 35
4.2.4 Ray casting techniques 36
4.2.5 Arm extension techniques 37
4.2.6 Image plane techniques 40
4.2.7 Other selection techniques 42
4.3 Selection in CHIMP and the Virtual Gorilla Exhibit 43
4.3.1 CHIMP 43
4.3.2 Virtual Gorilla Exhibit 44
5 Manipulation 45
5.1 Issues of manipulation 45
5.1.1 What is manipulation? 45
5.1.2 Constraints 46
5.1.3 Two-handed manipulation 47
5.2 Manipulation techniques 48
5.2.1 Direct manipulation 49
5.2.2 Tools and widgets 50
5.2.3 Implementing constraints 52
5.3 Manipulation in CHIMP and the Virtual Gorilla Exhibit 53
5.3.1 CHIMP 53
5.3.2 Virtual Gorilla Exhibit 55
6 Virtual menus 57
6.1 Issues of virtual menus 57
6.1.1 What are (virtual) menus? 57
6.1.2 Advantages of virtual menus 59
6.2 Implementation of virtual menus 59
6.2.1 Placement of menus 60
6.2.2 Dimensionality of virtual menus 60
6.2.3 The difficulty of selecting in VEs 62
6.3 Virtual menus in CHIMP and the Virtual Gorilla Exhibit 63
6.3.1 CHIMP 63
7 Discussion and conclusions 66
7.1 Virtual navigation 66
7.2 Virtual selection 67
7.3 Virtual manipulation 67
7.4 Virtual menus 68
7.5 Conclusions 68
8 References 69
TU Delft 10
1 Introduction
This report aims to give an overview of interaction techniques in virtual environments (VEs). The main focus will be on various software solutions that have been devised to allow the user to easily perform basic tasks in these environments. However, certain hardware issues will also be discussed, since these are often closely related to the software. This chapter provides a (very) brief overview of the field of virtual environment systems (also known as virtual reality). It starts of with a definition of the term virtual environments, after which two types of VE systems will be distinguished. A description of the basic components of a general virtual environment system is included as well. This is followed by a short overview of the various types of hardware devices that are currently being used with VE systems. Next, a classification of the different tasks for which VE systems have been used will be given (with some example applications). After that follows a discussion of the importance of a user being able to interact with the virtual environment. Then, an overview of the following chapters will be given. Finally, two sample applications will be introduced that will used throughout this report to illustrate several of the techniques discussed.
1.1 What are virtual environments?
A lot of different definitions exist for virtual environments and the virtual reality experience. One that covers the subject fairly well is the following [Barfield ‘95]:
A Virtual Environment (VE) is a computer-simulated world consisting of software representations of real (or imagined) agents, objects and processes; and a human-computer interface for displaying and interacting with these models.
There are a few things worth noting in this definition. First of all, the virtual environment has to be computer-generated. In a lot of texts teleoperator systems, where a robot or vehicle is controlled by a human from a (large) distance, are also categorized as being VE systems. However, these will not be considered in this report. Secondly, ‘displaying’ not only includes stimulation of the visual sense, but of the other senses as well, for example through the use of auditory or tactile displays. Another important thing is that the user should be able to interact with the models (see paragraph 1.3).
1.1.1 Types of VE systems
Besides the teleoperator systems mentioned above, two types of VE systems can be distinguished [Dani ‘96]:
· Immersive systems. The user’s field of view is completely surrounded by a synthetic, computer-generated, three-dimensional environment. Usually, the user is wearing some type of head-mounted display (HMD) and the user’s head is tracked to provide him with a perspective-correct view of the scene at all times, depending on the position and orientation of his head. Ultimately, all the sensory information received by the user is generated by a computer, thereby entirely shutting him off from the real world (see Figure 1.1).
· Non-immersive systems. The user typically wears a pair of (see-through) stereo glasses to view a three-dimensional scene on a normal display. With such a system, the user is able to see the real world as well as the computer-generated one. A normal computer monitor can be used as a (visual) display, but more advanced examples of this type of system include the Virtual Workbench and the CAVE system. A picture of the Virtual Workbench is shown in Figure 1.2.
TU Delft 10
Figure 1.1 Immersive system [WebUDW]
Figure 1.2 Virtual Workbench [WebSTANFORD]
TU Delft 10
Both types of systems mentioned consist of three basic components [Barfield ‘95]:
· Computer hardware and software to model and render the environment.
· A logical interface that specifies what parameters of the virtual environment and its models can be changed, and when.
· A physical interface consisting of one or more visual, haptic or auditory displays for presenting the virtual environment to the user, and for monitoring the actions of the user.
In general, the first component should be completely transparent to the user. The logical interface determines the ways in which the user can interact with the VE (e.g. it states that the user can change the position and the orientation of the viewpoint). The physical interface determines how the environment is presented to the user (e.g. by using a head-mounted display). It also determines how the user can perform the interactions as specified by the logical interface (e.g. a 3D tracker is attached to the user’s head to determine the position and orientation of the viewpoint). This report will focus on the logical interface.
1.1.2 General tasks in VE systems
Although VE systems have found applications in such diverse fields as engineering, entertainment, science, and training, four general tasks can be distinguished. These will be described and some example application areas will be given for each of them [Barfield ‘95][Vince ‘95].
· Off-line training and rehearsal
· On-line comprehension
· Off-line learning and knowledge acquisition
· On-line performance
In on-line training and rehearsal, the user typically practices a task in the VE system that he has to perform in the real world at a later time. Examples are flight simulators, surgery simulations, and military simulations. Obviously, mistakes can be fatal in these real-world tasks, so the ability to practice them in a virtual world is very convenient.
When on-line comprehension is the goal, the environment is used to reach understanding of and insight into the structure and properties of the data that is displayed or the workings of a particular process. Examples of this include interpretation of scientific data, and simulation of physical and chemical processes. Tasks such as architectural walkthroughs, whose purpose is evaluation of a design, belong to this category as well.
In off-line learning and knowledge acquisition tasks the user utilizes the VE to learn something, and to later apply that knowledge in a different, more abstract form. An example of this is the use of VE systems in education, where particular subjects can be illustrated (one could imagine students of a chemistry class being able to build complex molecules from basic atoms in a VE).
In contrast to the other three tasks, the goal of on-line performance is not to learn something. Instead, the actual actions the user performs are what matters. An example is the manipulation of objects in a remote, or non-viewable environment (such as an undersea robot), but this class also includes VR games.
In the next paragraph a short overview of the hardware devices that can be used to build the (physical) interface for these different types of systems will be given.
1.2 Hardware devices for VE systems
Various different types of hardware exist to provide user input to the computer, as well as output from the computer to the user. Because of the different (three-dimensional) nature of interaction in virtual environment systems, other devices besides the traditional mouse and keyboard have been developed to make use of these three dimensions for input and output. In this paragraph the general types of these devices will be discussed briefly, together with the way in which they can affect interaction in virtual environments. This list is not meant to be exhaustive, but serves to give an impression of the various techniques that are being used.
1.2.1 Output devices
Output devices serve to present the human senses of the user with information from the VE. This paragraph will briefly discuss visual and haptic output devices [DANI ‘96].
Visual devices
Visual output devices present the user with a visual image of the environment. These devices can generally be categorized in two classes: they are either immersive or non-immersive (see paragraph 1.1.1). A well known example of an immersive visual display is the head-mounted display (HMD), shown in Figure 1.3. An HMD shuts the user completely off from the real world and contains one
TU Delft 10
Figure 1.3 Head-mounted display [WebVRSYSTEMS]
Figure 1.4 Shutterglasses [WebSARA]
TU Delft 10
screen for each eye on which the graphics are displayed. Often, a HMD contains a pair of headphones for sound as well. In non-immersive displays the user typically wears a special pair of LCD (shutter) glasses (Figure 1.4) for looking at a screen on which the scene is displayed. This screen can be an ordinary monitor, but can also be a Virtual Workbench or a CAVE. To provide a stereo view, the glasses can be synchronized with the screen to alternately present a perspective-correct image to each eye, thus creating the illusion of depth.