High Fidelity Collaborative Virtual Environments

Mashhuda Glencross and Alan Chalmers 28th September 2005

Contents
1 2 3 Introduction Complex Interaction in Shared Virtual Environments Behaviourally-Rich Shared Virtual Environments 3.1 Advantages . . . . . . . . . . . . . . . . . . . . . 3.2 Requirements . . . . . . . . . . . . . . . . . . . . 3.2.1 High Fidelity Rendering . . . . . . . . . . 3.2.2 Multiple Interaction Modalities . . . . . . 3.2.3 Multiple Participants . . . . . . . . . . . . 3.2.4 Complex Simulation . . . . . . . . . . . . Perceptually Based Graphical Rendering 4.1 High Fidelity Graphics . . . . . . . . . . . . . . . 4.2 Image Quality Metrics . . . . . . . . . . . . . . . 4.2.1 Perceptually Based Image Quality Metrics . 4.3 Low-Level Perception-Based Error Metrics . . . . 4.3.1 Advanced Perception-Based Error Metrics . 4.3.2 Visible Differences Predictor . . . . . . . . 4.4 Comparing Real and Synthetic Images . . . . . . . 4.5 Selective Rendering . . . . . . . . . . . . . . . . . 4.5.1 Visual Perception . . . . . . . . . . . . . . 4.5.2 Peripheral Vision . . . . . . . . . . . . . . 4.5.3 Inattentional Blindness . . . . . . . . . . . 4.5.4 Task and Saliency Maps . . . . . . . . . . 4.5.5 Importance Maps . . . . . . . . . . . . . . 4.5.6 Selective Guidance System . . . . . . . . . 4.5.7 Map Generation . . . . . . . . . . . . . . 4.5.8 Results . . . . . . . . . . . . . . . . . . . 4.6 Perceptual Realism in Real-Time . . . . . . . . . . Interaction in Distributed Environments Effects of Haptic Feedback on Task-Performance 6.1 Single-User Case . . . . . . . . . . . . . . . . . . 6.2 Multi-User Case . . . . . . . . . . . . . . . . . . . Collaborative Experimental Studies 1 2 3 6 6 7 8 9 9 10 11 11 12 13 13 14 16 16 17 17 18 21 21 21 22 22 22 26 27 27 28 29

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . .

5 6

. . . . . . . . . . . . . . . . . . . .

10

11

12

13

14

15

7.1 Basdogan et al.s Wire Ring Experiment . . . . . . . . . . . . . . . . 7.2 Hubbolds Collaborative Stretcher Carrying . . . . . . . . . . . . . . 7.3 Sallns et al.s Cube Lifting Task . . . . . . . . . . . . . . . . . . . . Main Challenges 8.1 Shared Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Distribution over the Internet . . . . . . . . . . . . . . . . . . . . . . Classes of Interaction in Shared Virtual Environments 9.1 User-Entity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 User-User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 User-Entity-User . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Entity-Entity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Common Shared Interaction Modes 10.1 Turn Taking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Free-for-All . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Guided Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Distributing Simulations 11.1 Injection Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Users Interaction Information . . . . . . . . . . . . . . . . . . . . . 11.3 Synchronising State . . . . . . . . . . . . . . . . . . . . . . . . . . . Real Networks 12.1 Network Latency and Jitter . . . . . . . . . . . . . . . . . . . . . . . 12.2 Impact on Task Performance . . . . . . . . . . . . . . . . . . . . . . 12.3 Architectures for Distributed Systems . . . . . . . . . . . . . . . . . 12.3.1 Client-Server . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.2 Peer-to-Peer . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.3 Shades In-Between . . . . . . . . . . . . . . . . . . . . . . . Considerations for Behaviourally-Rich Distributed Haptic Environments 13.1 Techniques for Managing Latency and Jitter . . . . . . . . . . . . . . 13.2 Client-Server vs Peer-to-Peer . . . . . . . . . . . . . . . . . . . . . . 13.3 Rich Behaviour vs Haptic Response . . . . . . . . . . . . . . . . . . A Brief Survey of the State of the Art of Collaborative Haptic Environments 14.1 Peer-to-Peer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Client-Sever . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3 Hybrid Peer-to-Peer . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4 Hybrid Client-Server . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5 Commercial Solutions . . . . . . . . . . . . . . . . . . . . . . . . . Putting it All Together 15.1 Perceptually Based Distribution . . . . . . . . . . . . . . . . . . . . 15.2 Graphical, Behavioural and Haptic Correspondence . . . . . . . . . . 15.3 Providing Richer Haptic Response . . . . . . . . . . . . . . . . . . .

30 30 31 32 32 33 34 34 34 35 36 38 38 38 38 38 39 39 41 41 42 42 45 45 46 47 48 49 50 50 51 52 53 54 55 56 56 57 58 58

ii

Abstract The objective of this course is to describe the main challenges faced when building high delity engaging shared virtual environments that support complex behaviour and interaction. We will provide detailed discussions on techniques that can be adopted to facilitate this objective, by focussing on the demands of combining high quality graphics, better modes of interaction, rich behavioural simulations and appropriate distribution strategies. After introducing the eld of interaction and rich behaviour in collaborative virtual environments, we cover the main issues in three parts. First we consider techniques for improving the users experience by using high-delity graphical rendering, and explore how this may be achieved in real-time through exploitation of features of the human visual perception system. Second we examine also how additional sensory modalities such as audio and haptic rendering may further improve this experience. Finally we consider issues of distributing applications with a particular emphasis on avoiding potential pitfalls introduced by complex simulations, together with an analysis of real network conditions and the implications of these for distribution architectures that facilitate shared haptic interaction.

Introduction

The real-time performance requirement of interactive virtual environments has resulted in the rendering requirements generally dominating computational costs. Thus many virtual environment demonstrator applications constitute walk-throughs involving limited direct interaction with the environment. The real world however is complex, and contains many stimuli which simultaneously engage all our senses; sight, sound, touch, smell and taste. These combine with our experience of the world to give us an intuitive understanding of realism. Immersive virtual environments seek to provide a sense of being physically within a synthetically generated space, through a combination of multi-sensory stimuli, and isolating the user via the use of a head-mounted display. Unfortunately virtual reality technology is still cumbersome, the quality of head-mounted displays is still relatively poor, and the technology is still costly. Studies of presence [WS98,Sla99] including those which borrow from gaming theory, have shown that users need not be outtted with an immersive setup to become engaged in an environment [DH00, Man01a, BC04]. The level of engagement participants exhibit during game play, often using desktop computers or play stations, may be considered to be indicative of a sense of presence. A key factor contributing to this, is that during game play, users actively participate to carry out a particular task (usually shoot the bad guy). Typical rst person shooter games have a fast pace with the user having to consider other characters shooting at them, explosions, and sound effects while trying to achieve their objective. There is a lot of activity to occupy players leading to a high cognitive load. In a gaming context however a large number of assumptions based on the storyline can be made about the actions the player is most likely to perform, and therefore simplied simulation of the environment can be employed. On the other hand since it is impossible to envisage all the demands of every conceivable application, in the context of systems to build a wide variety of virtual environ1

ments, few such assumptions are possible. Consequently, it is not so straight forward to build such compelling and engaging environments. This leads to the question of how to make serious virtual reality applications more engaging and intuitive to use? The answer is to improve the correspondence of the virtual environment to the real world, through a combination of sophisticated interfaces for advanced human computer interaction coupled with good rendering and behavioural delity. In the following sections we describe what we mean by these.

Complex Interaction in Shared Virtual Environments

Despite a wide range of three-dimensional (3D) input devices being available (see Hand [Han97] for a good survey), the most common methods of interacting with a 3D environment is still via a standard mouse or a keyboard. Typically participants press keys and mouse buttons to invoke modal changes and use a standard mouse to provide positional information during drag and drop or navigation tasks. A possible reason for this may be the cost of more sophisticated input devices, their ease of use from a developers point of view, or perhaps even user familiarity. Due to a more intuitive spatial mapping between the real and virtual world 6 degree of freedom (DoF) input devices, such as a SpaceMouse or electromagnetically tracked devices, provide a more intuitive mode of interaction in 3D environments. However, participants often exhibit difculties with depth perception in virtual environments [RGA95]. A specic performance evaluation of input devices typically used in 3D applications was carried out by Roessler [RG98], and showed a paradoxical difference (which could be attributed to poor depth perception) between the actual and perceived accuracy of a tracked button used by test subjects. By engaging another sensory modality to support visual cues 3DoF and 6DoF force feedback (or haptic) devices could improve this situation, and in doing so enable better perception of the spatial layout of entities in 3D environments. Complex interaction is not purely related to input and output devices, but also potentially has a causal relationship to the state of entities with which the user interacts in the environment. This state change may be reported back to the user through multisensory feedback such as colour changes, and the use of auditory or touch (haptic) feedback. In a large number of virtual environments, participants may change simple attributes of entities. For example by selecting and moving a virtual representation of a baseball, its positional attributes will change. More complex entities which have physical characteristics allowing them to deform or ow, may not only have their position altered by user intervention but also their structure. Consequently, this in turn impacts upon the values provided to the algorithms used to compute the structure of the entity. Another possible source of complex interaction in virtual environments can be through communication with other remote participants. These may range from simply saying hello, to performing a complex sequence of tasks to achieve a shared goal. Few virtual environments employ many of these different types of complex interactions in conjunction with each other. Engaging environments should however ideally enable such interactions in behaviourally-rich environments, which may potentially contain a variety of dynamic entities whose state and/or structure may be changed. It is therefore 2

useful to consider how rich behaviour may be represented in frameworks for building virtual environments.

Behaviourally-Rich Shared Virtual Environments

Behaviour-rich virtual environments ideally model the Newtonian physics of the real world. However it is far beyond computational capabilities to faithfully simulate all the forces and torques acting on bodies, and correctly compute motion and deformations, for every entity in a complex virtual environment therefore ideally simulations need to be simplied [Gle00]. Provided such simplications can still be used to compute plausible or perceptually correct behaviour, it may be possible to dynamically simplify the underlying structure of the models according to run-time requirements [GHP01]. However, due to the performance demands of simulation and difculty of dynamic complexity management, scripted behaviours [CPGB94] are often invoked upon specic events or actions, for example a user picks up a specic entity which (when selected) records all the users activities until it is dropped (or unselected) [GFPB02]. This type of event based invocation of behaviour is a good way to represent a wide range of common background activities, such as a vehicle driving along a road or the bounce of a ball. Alternative approaches exploiting features are more intelligent, and determine the behavioural response of an entity from semantic information in the model. For example, a desk drawer can be opened and closed because it is a desk draw [KT99]. In prototyping or training applications however, if participants actions have an effect on the motion or structure of entities, scripted behaviour in itself is not sufcient. Feature based and constraint methods have been employed to perform rigid body assembly tasks however these methods require enough semantic information to be present in the models, and cannot simulate the deformation characteristics of complex exible components [MMF03]. The behavioural response in such applications may be specied through a number of rules and physically based simulation models. A variety of software frameworks exist to simplify the development of behaviourally-rich collaborative virtual environments [GB95, Hag96, PCMW00]. Many of these incorporate a sophisticated world model combined with spatial subdivision schemes, to limit the scope of particular characteristics. For readers wishing to know more about these, Pettifer presents a comprehensive survey [Pet99]. Pettifer et al.s [PCMW00] Deva system developed at the Advanced Interfaces Group at Manchester adopts essentially a client-server distribution architecture (detailed later in these notes), but with a exible conguration for communicating state changes of individual entities in a shared environment [Pet99]. The system achieves this by decoupling behaviour into an objective (semantic state) and subjective (perceptual state) component. The single objective part of entities resides on a server which maintains consistent state, and in order to either render or interact with the entities each participants client creates corresponding subjective parts. These possess behaviour and can act plausibly until otherwise instructed by the server, and so their state may subtly and briey differ from the objective reality as shown in Figure 1. It is the responsibility of the objective component to update at appropriate intervals all its corresponding subjects, and this information may be customised on a per-entity 3

Figure 1: The Deva object/subject model

Figure 2: Placeworld [PM01] basis [Pet99, Mar02]. Figures 2 to 5 and Figure 7 illustrate a number of example applications, implemented during large European funded projects, using the Deva system. The Placeworld application (Figures 2 and 3) was a world containing worlds, each individual world being an exhibit. It was developed for an art installation as part of a study into inhabited information spaces or electronic landscapes. Participants at the art show were able to explore each of the different exhibits, some of which had very different physical rules governing the behaviour of entities within them [PM01,Sha98,CP01]. They were also encouraged to contribute to the evolution of the landscape through the creation of hyper-links to favourite exhibits, and the use of personalised avatars [Mar02]. The QPit application [PCMT01] was implemented to visualise complex structured graph data, and employed a spring-mass-damper model to automatically congure its structure. By variations in the spring stiffness according to the different relationships between nodes (masses) in the data, the dynamics of the force model was used to determine an equilibrium conguration for the structure. This conguration presented a visualisation of the semantic information within the graph data. The Senet Game (shown in Figure 5) was an educational distributed environment designed to give children experience of a board game found in the ruins of the ancient Egyptian city of Kahun [EPMW99], and through this, enable a study of how to better design social learning environments. The application used bespoke avatars (representing an adult, young girl and boy) together with a radiosity rendition of an Egyptian

Figure 3: Placeworld legible city [Sha98, CP01]

Figure 4: QPit [PCMT01] room, complete with the game board, movable pieces, die, and kartouches on the wall containing the games instructions [Mar02]. Players took turns to throw the die and move their pieces according to the games rules; the aim being to be the rst to remove all their pieces from the board. The game itself was played by pairs of children, supervised in the environment by a teacher. A chat mechanism was used, which maintained a log (for later analysis) of all conversations that had taken place during a particular session of the game. Participants communicated by typing into an entry box that was part of the user interface. The game enabled participants to both co-operate to learn the rules, and compete to win [Mar02]. While a range of virtual reality applications do contain simulation and rich behaviour to varying degrees, these are commonly implemented as a part of bespoke applications. The Deva framework has demonstrated that with an appropriate infrastructure for complex behaviour supported as core functionality, it is possible to rapidly develop and customise the behaviour for a range of diverse applications. Physically based simulation techniques provide the most reliable method for computing realis-

Figure 5: Senet game [EPMW99] tic behaviour but often at a computational cost [Gle00]. Consequently these types of simulations are most often found in animation and computer aided design (CAD) applications, which impose less stringent real-time demands. Methods to manage the complexity of underlying simulations, while maintaining a consistent perception of the simulation offer the potential for wider scale adoption of simulation in real-time applications [GM98, GM99, GM00, GHP01].

3.1

Advantages

Rich behaviour in virtual environments can lead to highly engaging applications ranging from systems to prototype and maintain complex CAD designs, training, and entertainment applications. A particular example of a compelling and engaging application is VRSims welding simulator (shown in Figure 6). Welding is an especially dangerous occupation and many trainees suffer serious burns during the early parts of their vocational training. VRSims application aims to mitigate the level of injury through an immersive training setup prior to actual hands-on experience [VRS05]. The application combines tools very similar to real welding equipment. Users wear a custom head-mounted display which is tted into a welders mask, graphical, auditory and haptic feedback (using an FCS HapticMASTER) is supplied to closely approximate multisensory informational cues in the real task. Good real-world correspondence between the training conguration, and the actual task may account for the good level of skills transfer reported by VRSim [Wal05]

3.2

Requirements

In order to maximise real world correspondence, and in doing so build compelling virtual reality applications, we argue that it is necessary to have the following: Good quality graphics Complex interaction preferably engaging multiple sensory modalities Realistic simulation 6

Figure 6: Virtual welding simulator Image courtesy of FCS Control Systems, The Netherlands The level to which each of these factors are employed depends largely on the application. Good quality graphics may sufce for CAD applications, while high delity graphics may be required for interactive visualisations of architectural designs. Graphical and audio feedback may be sufcient for walk-throughs or simple visualisations, while haptic feedback may be required to assist in assembly tasks. Simple ocking behaviour [Rey87] may sufce for describing the behaviour of birds in synthetic landscapes, while more complex nite element models may be required to determine the deformation characteristics of materials in crash simulations [Win81, KSE 97, BFL 01]. With each of the above requirements for good real world correspondence imposing their own specic performance demands on applications, a particular challenge is to maximise the level of engagement within real-time performance constraints. Some of the techniques discussed in this tutorial may provide a mechanism for achieving this goal. 3.2.1 High Fidelity Rendering

High delity rendering is a term used to describe highly realistic graphical rendering. An important factor contributing to the realism of a computer generated image is the lighting. Illumination in the real-world is a complex process involving both natural and man-made light sources. We see shadows, reections, optical effects caused by refraction, perspective effects, and we perceive depth through subtle visual cues. The weather affects visibility, clouds exhibit complex self shadowing and illumination characteristics. A huge range of techniques exist to simulate and compute realistic lighting effects [Gla89, Hec92, CWH93, SP94, Jen01, WKB 02, CDR02]. However, realistic illumination covering the full range of complex processes in the real world is still a signicant challenge. Even more challenging is to achieve realistic illumination in real-time. Few metrics for rendering delity exist and much of the current work has been carried out in immersive contexts [MTHC03, MR04]. Measures of rendering delity in these studies are normally arrived at by an analysis of the subjective preferences

Figure 7: Collaborative CAD prototyping application [MGP 04] of participants. However, interestingly, these studies indicate that improved rendering delity could improve spatial task performance through better depth perception. 3.2.2 Multiple Interaction Modalities

The most common modality used to provide feedback in virtual environments is the visual one, colour changes are often used to indicate collisions or semantic information relating to tasks. Different graphical rendering styles can also be employed to convey information about entities in virtual environments. For example in a distributed CAD prototyping (Divipro) application, implemented using Deva, [MGP 04] and shown in Figure 7 a red highlight is used to visually indicate that an assembly constraint has been found. Part of the assembly is semi-transparent indicating it is semantically less important to the assembly task and merely provides contextual information. Audio cues were used in this application to provide feedback relating to collisions, geometric constraints, task and collaboration state. A range of different tunes were played to indicate if a particular constraint had been matched, was active, was met or broken. Subtle changes in background colour were used to inform participants of transfer of control between them, in order to mediate collaboration. Informal observations of assembly sequence completion times showed that these cues improved the users ability to perform assemblies. Since the application allowed multiple participants to collaborate on a given assembly, audio feedback was also provided to give information relating to allowable interactions, for example when it was acceptable for other users to partake in a sequence. Clearly audio cues could be incorporated into many virtual environment applications, to convey task related semantic information, but it is not the only alternative modality available. The addition of haptic feedback offers a more engaging and tightly coupled interaction mechanism, in the sense that the same device is used both for input and display with each reinforcing the other. Due to limitations in both in the technology and soft8

ware, most haptic devices currently available are unable to convey a real sense of touch as it is experienced in the real world. However, a sense of contact with solid or viscous materials (which resist to varying degrees) penetration can be conveyed. In the Divipro application, haptic feedback was used to provide force responses relating to collisions and locking to constraints. Again, informally, it was found to be a compelling addition to support performance of assembly tasks. Similarly, in the welding application, the use of haptic feedback conveys information about the position and material properties of the weld. A nal modality often exploited in re-creations of the past for museum exhibits (for example the Jorvik Viking Museum [Jor05]) is the sense of smell (or olfactory feedback). However, currently it is not practical to use this in virtual environments. For certain kinds of training tasks such as the welding simulator, it could be a useful modality to exploit as it would again convey additional information relating to the state of the material being used in the weld. 3.2.3 Multiple Participants

In the real world our senses are not only stimulated by interaction with entities in the environment, but also through interaction with other people. Multiple participants in virtual environments contribute to the realism and level of engagement an application can offer. Environments populated by synthetic characters soon become boring and predictable, as users quickly recognise any preprogrammed sequences. Real participants customise interactions in virtual environments through their own personal choices, and while these may be predictable (if you know the person) they are not always the same. Evidence for the suggestion that multiple participants improves the level of engagement in virtual environments can be found in the recent popularity of online multi-player gaming. In particular studies of communication between participants have shown that intuitive mechanisms improve the social experience of players and the level of engagement [HRF03] In addition to communicating and interacting with each other, multiple participants can also choose to collaborate to achieve a shared objective. This phenomenon is also seen in multi-player online games, where friends collaborate to take out participants they do not consider to be in their social group [Man01b]. In a number of training and assembly or maintenance tasks, collaborative interaction may also be essential to completing the task. However in a shared behaviour-rich environment it is essential that each participant sees a plausible representation of the environment, and that the consistency of this is always maintained [Pet99, Mar02]. As we will explain in this tutorial, this can be non-trivial. 3.2.4 Complex Simulation

Complex simulations that application developers may typically wish to incorporate into virtual environments can be classied as follows: Emergent behaviour Material properties 9

 Natural phenomena Emergent behaviour involves a number of mainly independent entities each conforming to some simple particle physics, scripted responses, or ad-hoc goals. Combined, the behaviour of these entities leads to a complex overall response. This type of simulation is used to support ocking behaviour [Rey87,BCN97], and crowd simulations in multiuser [BGL97, MBCT98] and evacuation scenarios [UT01, UT02]. More complex numerical simulation techniques are used to compute material properties and motion of particles, rigid-body, articulated-rigid-body, and deformable systems. Often these employ numerical solvers to solve equations of motion and deformation characteristics of nite elements, or surface tting methods to compute iso-surfaces from force eld functions (implicit surfaces [BW97]). These techniques are commonly used in animations for example, to simulate soft substances [DG95, CGD97], deformable objects [TF88, TF98], cloth draping behaviour [BHW94, BW98, BWK03], and in applications such as crash [Win81, KSE 97, EMTTT98, BFL 01] and surgical simulations [BN98, CDA99, DCA99, BHS01, WDGT01]. However, the performance demands of many of these methods have meant that few have found use in many real-time virtual reality applications. Simulating natural phenomena such as wind, smoke, re [SF93, Sta00, FSJ01], clouds [Bli82, MYDN01], aurora [BRS 03] and water [FR86, KWF 01, EMF02] is an active research area and a large variety of traditional procedural methods [MPPW94], uid dynamic simulations, and modern implementations employing the graphical processor unit (GPU) exist [HCSL02, HL01, HBSL03, Har04, LLW04]. Many of these methods achieve visually impressive results in real-time, with research into faster and more realistic simulations ongoing. Illumination techniques have also been developed to incorporate optical effects such as rainbows, cloud-bows, sun-dogs [Mus89], atmospheric mirages [KH97] and many other subtle lighting effects commonly seen in the real world. However combining simulation, rendering, realistic illumination, complex interaction, and multiple participants in real-time environments is a signicant computational challenge. Techniques to selectively compute and render perceptually important parts of the environment, by exploiting the effects of high cognitive load during task performance, is essential to achieving this in real-time. In the following sections, we will elaborate on this.

Perceptually Based Graphical Rendering

The computer graphics industry, and in particular those involved with lms, games, simulation and virtual reality, continue to demand more realistic computer generated images, that is synthesised images that more accurately match the real scene they are intended to represent. Despite the ready availability of modern high performance graphics cards, the complexity of the scenes being modelled and the high delity required of the images means that rendering such images is still simply not possible in a reasonable, let alone real-time on a single computer. Two approaches may be considered in order to achieve such realism in real-time: Parallel Processing and Visual

10

Perception guided methods. Parallel Processing has a number of computers working together to render a single image, which appears to offer almost unlimited performance, however, enabling many processors to work efciently together is a signicant challenge [CDR02, PMS 99, WSBW01]. Visual Perception, on the other hand, takes into account that it is the human who will ultimately be looking at the resultant images, and while the human eye is good, it is not perfect. As we will see in this section, exploiting knowledge of the human visual system can indeed save signicant rendering time by simply not computing detail in those parts of a scene that the human will fail to notice.

4.1

High Fidelity Graphics

Realness - the state of being actual or real. Obviously this denition refers to the real world and our perception of it, however frequently in the doctrine of computer science the terms realistic, realism and real are discussed. Obviously anything represented on a computer is not real but just an approximation, so what do these expressions refer to? There are many uses for computers in the world we live in ranging from high performance games to high accuracy mathematical calculations. Both of these examples and countless more have one thing in common the need to have some level of realism. Within the games industry it is important for there be some link with reality (or at least some conceivable fantasy of reality) to involve the player in the game. However the level of realism needed in a computer game is related to the genre and objective of the game. At the other end of the spectrum there exist applications that directly apply to the real world; one example might be a software package that is employed to perform aerodynamics calculations during the design process of a new ghter aircraft. In this circumstance an extremely high delity simulation of reality is required to ensure that the plane will y. However within the context of a computer game it is more important that the plane looks realistic and behaves as expected, while in the design application the appearance of the plane is less critical (and may not even be presented) but realistic behaviour is crucial to the application.

4.2

Image Quality Metrics

Reliable image quality assessments are necessary for the evaluation of realistic image synthesis algorithms. Typically the quality of the image synthesis method is evaluated using image-to-image comparisons. Often comparisons are made with a photograph of the scene that the image depicts. Several image delity metrics have been developed whose goals are to predict the amount of differences that would be visible to a human observer. It is well established that simple approaches like mean squared error do not provide meaningful measures of image delity, thus more sophisticated measures which incorporate a representation of the human visual system are needed. It is generally recognised that more meaningful measures of image quality are obtained using techniques based on visual (and therefore subjective) assessment of images, after all most nal uses of computer generated images will be viewed by human observers.

11

4.2.1

Perceptually Based Image Quality Metrics

A number of experimental studies have demonstrated many features of how the human visual system works. However, problems arise when trying to generalise these results for use in computer graphics. This is because, often, experiments are conducted under limited laboratory conditions and are typically designed to explore a single dimension of the human visual system. Instead of reusing information from these previous psychophysical experiments, new experiments are needed which examine the human visual system as a whole rather than trying to probe individual components. Using validated image models that predict image delity, programmers can work toward achieving greater efciencies in the knowledge that resulting images will still be faithful visual representations. Also in situations where time or resources are limited and delity must be traded off against performance, perceptually based error metrics could be used to provide insights into where corners could be cut with least visual impact. Using a simple ve sided cube as their test environment Meyer et al. [MRC 86] presented an approach to image synthesis comprising separate physical and perceptual modules. They chose diffusely reecting materials to build a physical test model. Each module was veried using experimental techniques. The test environment was placed in a small dark room. Radiometric values predicted using a radiosity lighting simulation were compared to physical measurements of the radiant ux density in the real scene. Results showed that irradiation was greatest near the centre of the open side of the cube. This area provided the best view of the light source and other walls. In summary, there was a good agreement between the radiometric measurements and the predictions of the lighting model. Rushmeier et al. [RLP 95] explored using perceptually based metrics, based on image appearance, to compare image quality to a captured image of the scene being represented. The goal of this work was to obtain results by comparing two images using models that give a large error when differences exist between images. The following models attempt to model effects present in the human visual system. Each uses a different Contrast Sensitivity Function (CSF) to model the sensitivity to spatial frequencies. Model 1 After Mannos and Sakrison: First, all the luminance values are normalised by the mean luminance. The non linearity in perception is accounted for by taking the cubed root of each normalised luminance. A Fast Fourier Transform (FFT) is computed of the resulting values, and the magnitudes of the resulting values are ltered with a CSF to an array of values. Finally the distance between the two images is computed by nding the Mean Square Error (MSE) of the values for each of the two images. This technique therefore measures similarity in Fourier amplitude between images. Model 2 After Gervais et al: This model includes the effect of phase as well as magnitude in the frequency space representation of the image. Once again the luminances are normalised by dividing by the mean luminance. An FFT is computed producing an array of phases and magnitudes. These magnitudes are then ltered with an anisotropic CSF lter function constructed by tting splines to psychophysical data. 12

Model 3 After Daly: In this model the effects of adaptation and non-linearity are combined in one transformation, which acts on each pixel individually. In the rst two models each pixel has signicant global effect in the normalisation by contributing to the image mean. Each luminance is transformed by an amplitude nonlinearity value. An FFT is applied to each transformed luminance and then they are ltered by a CSF (computed for a level of 50 cd/m2). The distance between the two images is then computed using MSE as in model 1. The Visible Difference Predictor (VDP) is a perceptually based image quality metric proposed by Daly [Dal93]. Myskowski realised this metric had many potential applications in realistic image synthesis [Mys98]. He completed a comprehensive validation and calibration of VDP response via human psychophysical experiments. The VDP was tested to determine how close predictions come to subjective reports of visible differences between images by designing two human psychophysical experiments. Results from these experiments showed a good correspondence for shadow and lighting pattern masking and in comparison of the perceived quality of images generated as subsequent stages of indirect lighting solutions.

4.3

Low-Level Perception-Based Error Metrics

Perceptual error metrics have also been used in several other areas. Gibson and Hubbold [GH97] proposed a perception-driven hierarchical algorithm for radiosity used to decide when to stop hierarchy renement. Links between patches are not re-ned anymore once the difference between successive levels of elements becomes unlikely to be detected perceptually. Gibson and Hubbold also applied a similar error metric to measure the perceptual impact of the energy transfer between two interacting patches, and to decide upon the number of shadow feelers that should be used in visibility test for these patches. Perceptually-informed error metrics have also been successfully introduced to control the adaptive mesh subdivision and mesh simplication. Specic implementations have been performed and analysed by Myszkowski [Mys98], Gibson et al. [GCHH03] 28, and Volevich et al. [VMKK00]. 4.3.1 Advanced Perception-Based Error Metrics

The scenario of embedding advanced human visual system models into global illumination and rendering algorithms is very attractive, because computation can be perceptiondriven specically for a given scene. Bolin and Meyer [BM98] developed an efcient approximation of the Sarnoff Visual Discrimination Model (VDM), which made it possible to use this model to guide samples in a rendered image. Because samples were only taken in areas where there were visible artifacts, some savings in rendering time compared to the traditional uniform or adaptive sampling were reported. Myszkowski [Mys98] has shown some applications of the VDP to drive adaptive mesh subdivision taking into account visual masking of the mesh-reconstructed lighting function by textures. Ramasubramanian et al. [RPG99] have developed their own image quality metric which they applied to predict the sensitivity of the human observer to

13

noise in the indirect lighting component. This made possible more efcient distribution of indirect lighting samples by reducing their number for pixels with higher spatial masking (in areas of images with high frequency texture patterns, geometric details, and direct lighting variations). All computations were performed within the framework of the costly path tracing algorithm, and a signicant speedup of computations was reported compared to the sample distribution based on purely stochastic error measures. A practical problem arises that the computational costs incurred by the human visual system models introduce an overhead to the actual lighting computation, which may become the more signicant the more rapid is the lighting computation. This means that the potential gains of such perception-driven computation can be easily cancelled by this overhead depending on many factors such as the scene complexity, performance of a given lighting simulation algorithm for a given type of scene, image resolution and so on. The human visual system models can be simplied to reduce the overhead, e.g., Ramasubramanian et al. [RPG99] ignore spatial orientation channels in their visual masking model, but then underestimation of visible image artifacts becomes more likely. To prevent such problems and to compensate for ignored perceptual mechanisms, more conservative (sensitive) settings of the human visual system models should be applied, which may also reduce gains in lighting computation driven by such models. 4.3.2 Visible Differences Predictor

Although, substantial progress in physiology and psychophysics studies has been achieved in recent years, the human visual system as the whole, and in particular, the higher order cognitive mechanisms, are not fully understood. Only the early stages of the visual pathway beginning with the retina and ending with the visual cortex are considered as mostly explored. It is believed that the internal representation of an image by cells in the visual cortex is based on spatial frequency and orientation channels. The channel model provides a good explanation of visual characteristics such as: The overall behavioural Contrast Sensitivity Function (CSF) - visual system sensitivity is a function of the spatial frequency and orientation content of the stimulus pattern. Spatial masking - detect ability of a particular pattern is reduced by the presence of a second pattern of similar frequency content. Sub-threshold summation - adding two patterns of sub-threshold contrast together can improve detect ability within a common channel. Contrast adaptation - sensitivity to selected spatial frequencies is temporarily lost after observing high contrast patterns of the same frequencies. The spatial frequencies after effects - as result of the eye adaptation to a certain grating pattern, other nearby spatial frequencies appear to be shifted. Because of these favourable characteristics, the channel model provides the core of the most recent human visual system models that attempt to describe spatial vision. The 14

Figure 8: Block diagram of the Visible Differences Predictor (heavy arrows indicate parallel processing of the spatial frequency and orientation channels) VDP is considered one of the leading computational models to predicting the differences between images that can be perceived by the human observer. The VDP receives as input a pair of images, and as output it generates a map of probability values, which characterise perceptibility of the differences. The input target and mask images undergo an identical initial processing, as shown in Figure 8. At rst, the original pixel intensities are compressed by the amplitude non-linearity based on the local luminance adaptation, simulating Webers law-like behaviour. Then the resulting image is converted into the frequency domain and processing of CSF is performed. The resulting data is decomposed into the spatial frequency and orientation channels using the Cortex Transform, which is a pyramid-style, invertible, and computationally efcient image representation. Then the individual channels are transformed back to the spatial domain, in which visual masking is processed. For every channel and for every pixel, the elevation of detection threshold is calculated based on the mask contrast for that channel and that pixel. The resulting threshold elevation maps can be computed for the mask image, or mutual masking can be considered by taking the minimal threshold elevation value for the corresponding channels and pixels of the two input images. These threshold elevation maps are then used to normalise the contrast differences between target and mask images. The normalised differences are input to the psychometric function which estimates probability of detecting the differences for a given channel. This estimated probability value is summed across all channels for every pixel. Finally, the probability values are used to visualise visible differences between the target and mask images. It is assumed that the difference can be perceived for a given pixel when the probability value is greater than 0.75, which is standard threshold value for discrimination tasks. When a single numeric value is needed to characterise the differences between images, the percentage of pixels with probability greater than this threshold value is reported. The main advantage of the VDP is a prediction of local differences between images (on the pixel level). The original Daly model also has some disadvantages, for example, it does not process chromatic channels in input images. However, in global illumination applications many important effects such as the solution convergence or the quality of shadow reconstruction can be relatively well captured by the achromatic mechanism, which is far more sensitive than its chromatic counterparts. The VDP seems to be one of the best existing choices for the prediction of image quality for various settings of global illumination solutions.

15

Figure 9: Results obtained by McNamara et al. in lightness matching task experiments

4.4

Comparing Real and Synthetic Images

A number of experiments have been conducted at the University of Bristol where comparisons have been made between real and synthetic images. These comparisons although comparing real and synthetic images have been task specic and have employed only simple controlled environments. McNamara [MCTG00], performed a series of experiments where subjects were asked to match lightness patches within the real world to those on a VDU. They discovered that a photograph of the real scene gave the highest perceptual match, with a high quality tone mapped rendered version coming a close second. A graph of their ndings is shown in Figure 9. In all cases (apart from the ray tracing and radiosity results) Radiance was used to render the images.

4.5

Selective Rendering

The perception of a virtual environment depends on the user and where he/she is currently looking in that environment. Visual attention is the process by which we humans select a portion of the available visual information for localisation, identication and understanding of objects in an environment. It allows our visual system to process visual input preferentially by shifting attention about an image, giving more attention to salient locations and less attention to unimportant regions [Dal93, IKN98, YPG01]. When attention is not focused onto items in a scene they can literally go unnoticed [CCL02, Yar67]. So whilst visual attention may not be appropriate for, for example, a completed computer generated lm which will be watched by many viewers simultaneously, it can certainly assist in the production of the lm when the developers are focusing on particular aspects of a scene, for example how the movement of a character affects the lighting in a particular region. The key to achieving realism in real-time in virtual environments on current technology is knowing where the user will be looking in the image and rendering these areas at a very high quality, while the remainder of the scene, not attended to by the

16

user, can be rendered to a much lower quality without the user being aware of the quality difference [YPG01, CCL02, CKM03]. For a surprisingly large number of applications, high level task maps and low level saliency maps can indeed indicate where the user will be looking with complete accuracy. We thus dene a perceptually realistic scene as one in which the viewer perceives to be of a very high quality, where in fact signicant parts of the image may be rendered at a much lower quality. 4.5.1 Visual Perception

Visual attention is a coordinated action involving conscious and unconscious processes in the brain, which allow us to nd and focus on relevant information quickly and efciently. If detailed information is needed from many different areas of the visual environment, the eye does not scan the scene in a raster-like fashion, but jumps so that the relevant objects fall sequentially on the fovea. These jumps are called saccades [Yar67]. There are two general visual attention processes, labelled bottom-up and top-down, which determine where humans locate their visual attention [Jam90]. The bottom-up process is purely stimulus driven, for example, a re in the dark, a sudden movement, a red apple in a green tree, or the lips and eyes of another person - the most mobile and expressive elements of a face. In all these cases, the visual stimulus captures attention automatically without volitional control. This is evolutionary; the movement may be danger lurking behind a bush, or we may need to nd ripe fruit for our meal. In contrast, the top-down process is under voluntary control, and focuses attention on one or more objects that are relevant to the observers goal when studying a scene. Such goals might include looking for a lost child, searching for an exit, or remembering the position of the people and objects in a room. General knowledge of the human visual system has been widely used to improve the quality of rendered images [FPSG96,GTS 97,MCTG00,MTAS01,PFG98,RPG99]. Other approaches have focused on how complex model detail can be reduced without any reduction in the viewers perception of the environment [LH01, LRW 01, MS95, Red97, BFM01]. The application of visual attention models in computer graphics has so far mainly exploited only peripheral vision and the bottom-up visual attention process. Recent approaches however have combined both the top-down and bottom-up processes [SDL 05]. 4.5.2 Peripheral Vision

Due to the fact that the human eye only processes detailed information from a relatively small part of the visual eld, it is possible to reduce detail in the periphery without upsetting visual processing. In numerous studies, McConkie and Loschky [ML97, LM99, LMYM01] used an eye-linked, multiple resolution display that produces high visual resolution only in the region to which the eyes are directed. They were able to show that photographic images ltered with a window radius of 4.1 r produced results statistically indistinguishable from that of a full, high-resolution display. The display they propose does, however, encounter the problem of updating the multi-resolution image after an eye movement without disturbing the visual processing. Their work has

17

shown that the image needs to be updated after an eye saccade within 5 milliseconds of a xation, otherwise the observer will detect the change in resolution. These high update rates were only achievable using an extremely high temporal resolution eye tracker, and pre-storing all possible multi-resolution images that were to be used. In another experiment, Watson et al. [1997] evaluated the effectiveness of high detail insets in head-mounted displays. The high detail inset they used was rectangular and was always presented at the nest level of resolution. Three inset conditions were investigated: a large inset - half the complete displays height and width, a small inset size - 30% of the complete displays height and width, and no inset at all. The level of peripheral resolution was varied at: ne resolution 320 240, medium resolution 192 144 and coarse resolution 64 48. Their results showed that although observers found their search targets faster and more accurately in a full high resolution environment, this condition was not signicantly better than the high-resolution inset displays with either medium or low peripheral resolutions. 4.5.3 Inattentional Blindness

In 1967, the Russian psychologist Yarbus recorded the xations and saccades observers made while viewing natural objects and scenes. Observers were asked to answer different questions concerning the scene in Repins depiction of An Unexpected Visitor [Yar67]. This resulted in substantially different saccade patterns, each one being easily construable as a sampling of those picture objects that were most informative for the answering of the questions, as shown in Figure 10. The failure of the human to see unattended items in a scene, is known as inattentional blindness [MR98, SC99]. The concept of task maps, which are two dimensional maps highlighting the task at hand, has recently been introduced to exploit this topdown approach [CCW03]. Previously, Cater et al. [CCL02] showed that the conspicuous objects in a scene that would normally attract the viewers attention can be deliberately ignored if they are irrelevant to the task at hand. The effectiveness of inattentional blindness in reducing overall computational complexity was investigated by asking a group of users to perform a specic task: to watch two animations and in each of the animations, count the number of pencils that appeared in a mug on a table in a room as he/she moved on a xed path through four such rooms. In order to count the pencils, the users needed to perform a smooth pursuit eye movement tracking the mug in one room until they have successfully counted the number of pencils in that mug and then perform an eye saccade to the mug in the next room. The task was further complicated and thus retained the viewers attention, by each mug also containing a number of spurious paintbrushes. The study involved three rendered animations of an identical y through of four rooms. The only difference being the quality to which the individual animations had been rendered. The three qualities of animation were: High Quality(HQ): Entire animation rendered at the highest quality. Low Quality(LQ): Entire animation rendered at a low quality with no anti-aliasing.

18

(a) Repins picture

(b) Eye movements

Figure 10: The effects of a task on eye movements. Repins picture was examined by subjects given one of six different instructions; 1. Free viewing, 2. Judge their ages, 3. Guess what they had been doing before the unexpected visitors arrival, 4. Remember the clothes worn by the people, 5. Remember the position of people and objects in the room & 6. Estimate how long the unexpected visitor has been away from the family [Yar67] Circle Quality(CQ): Low Quality Picture with high quality rendering in the visual angle of the fovea (2 degrees) centred around the pencils, shown by the inner green circle in Figure 11(a). The high quality is blended to the low quality at 4.1 degrees visual angle (the outer red circle in gure 11(a)). Each frame for the high quality animation took on average 18 minutes 53 seconds to render on a Intel Pentium 4 1GHz Processor, while the frames for the low quality animation were each rendered on average in only 3 minute 21 seconds. A total of 160 subjects were studied which each subject seeing two animations of 30 seconds each displayed at 15 frames per second. Fifty percent of the subjects were asked to count the pencils in the mug while the remaining 50% were simply asked to watch the animations. To minimise experimental bias the choice of condition to be run was randomised and for each, 8 were run in the morning and 8 in the afternoon. Subjects had a variety of experience with computer graphics and all exhibited at least average corrected vision in testing. A count down was shown to prepare the viewers that the animation was about to start followed immediately by a black image with a white mug giving the location of the rst mug. This ensured that the viewers focused their attention immediately on the rst mug and thus did not have to look around the scene to locate it. On completion of the experiment, each participant was asked to ll in a detailed questionnaire. This questionnaire asked for some personal details, including age, occupation, sex and level of computer graphics knowledge. The participants were 19

(a) Visual angle covered by the fovea for mugs in the rst two rooms at 2 degrees (green circles) and 4.1 degrees (red circles)

(b) Experimental results for the two tasks: Counting the pencils and simply watching the animations

Figure 11: Results of an experiment to assess the range of the region of interest for high quality rendering while performing a task and observing and animation then asked detailed questions about the objects in the rooms, their colour, location and quality of rendering. These objects were selected so that questions were asked about objects both near the foveal visual angle (located about the mug with pencils) and in the periphery. They were specically asked not to guess, but rather state dont remember when they had failed to notice some details. Figure 3(b) shows the overall results of the experiment. Obviously the participants did not notice any difference in the rendering quality between the two HQ animations (they were the same). Of interest is the fact that, in the CQ + HQ experiment, 95% of the viewers performing the task consistently failed to notice any difference between the high quality rendered animation and the low quality animations where the area around the mug was rendered to a high quality. Surprisingly 25% of the viewers in the HQ+LQ condition and 18% in the LQ+HQ case were so engaged in the task that they completely failed to notice any difference in the quality between these very different qualities of animation. Furthermore, having performed the task of counting the pencils, the vast majority of participants were simply unable to recall the correct colour of the mug (90%) which was in the foveal angle and even less the correct colour of the carpet (95%) which was outside this angle. The failure to notice was even higher for less obvious objects, especially those outside the foveal angle. Overall the participants who simply watched the animations were able to recall far more detail of the scenes, although the generic nature of the task given to them precluded a number from recalling such details as the colour of specic objects, for example 47.5% could not recall the correct colour of the mug and 53.8% the correct colour of the carpet. The results of this work demonstrated that when observers were performing the task within an animation, their visual attention was xed exclusively on the area of the task and they consistently failed to notice the signicant difference in rendering quality 20

between the two animations. Of course, knowledge of the task being performed was fundamental in determining where a view was attended. For many applications, for example, lm production, games and simulators, such knowledge exists. 4.5.4 Task and Saliency Maps

Low-level saliency models determine what visual features will involuntarily attract our attention in a scene. Visual psychology researchers such as Yarbus [Yar67], Itti and Koch [IK00] and Yantis [Yan96] showed that the visual system is highly sensitive to features such as edges, abrupt changes in colour, and sudden movements. This low-level visual processing has been exploited in computer graphics by Yee et al. [Yee00, YPG01] to accelerate animation renderings with global illumination, by applying a model of visual attention to identify conspicuous regions. Yee constructs a spatiotemporal error tolerance map, called the Aleph map, from spatiotemporal contrast sensitivity and a low-level saliency map, for each frame in an animation. The saliency map is obtained by combining the conspicuity maps of intensity, colour, orientation and motion. Subsequent work by Marmitt and Duchowski [MD02] showed, however, that care must be taken with bottom-up visual attention models as they do not always predict attention regions in a reliable manner. 4.5.5 Importance Maps

Sundstedt el al [SDL 05] have a developed a rendering framework that exploits visual attention processes, in order to selectively render high-delity animations in a reasonable time, while maintaining perceived quality. The framework is composed of two major processes: selective guidance uses a combination of saliency and a measure of task relevance to direct the rendering computation. selective rendering corresponds to the traditional rendering computation. However, computational resources are focused on parts of the image which are deemed more important by the selective guidance. 4.5.6 Selective Guidance System

Sundstedt et al.s selective guidance system produces an importance map which is used to direct the rendering process. An importance map, IM(wt,ws,op), is a twodimensional map representing image space. The values within the map dictate where computational resources are best spent to obtain the highest perceptual result, whereby the highest values will result in preferential rendering. The importance map is a combination of a task map to model the effect of top-down visual attention and a saliency map to model bottom-up visual attention. The parameters in the importance map allow the user to specify a weighting that denes the relative importance of the task map and the saliency map. The parameter wt is a coefcient which is applied to the values in

21

the task map. The other coefcient ws is applied to the values in the saliency map, and the two terms are combined through the operator op. Selection of an appropriate operator controls the combination of the two maps in the selective renderer. The implementation currently uses addition to combine the information from both maps such that all weighted features are preserved. 4.5.7 Map Generation

The process begins with a rapid image estimate (in the order of ms) of the scene using a quick rasterisation pass in hardware [LDC05]. This estimate can be used in two ways. Firstly for building the task map by identifying user-selected task objects, and secondly, by using it as an input to a saliency generator. In the creation of the task map the program reads in the geometry information and a list of predened task objects. It then produces a map with task objects in white and the other geometry in black. A fovealangle gradient is then applied around task-related objects in the task map [SCCD04]. In the creation of the saliency map the image estimate serves to locate areas where an observer will be most likely to look. Figure 12 shows the various maps for the reference image 14(a). Sub-gures 12(a) and 12(b), in Figure 12 show the task map with and without a foveal-angle gradient added. Sub-gure 12(c) demonstrates the saliency map, while 12(d) shows the combination of the task map and the saliency map with equal weighting. 4.5.8 Results

To test their importance maps, Sundstedt et al. rendered ten animations of a corridor scene [SDL 05] shown in Figure 14. They rendered a high quality (HQ), low quality (LQ), task map quality (TQ), saliency map quality (SQ) and a combined task map and saliency map quality (IQ) animation. Table 1 shows the conditions tested and some of the maximum and minimum rendering parameters values used for the renderings. The animations were all rendered at 900 900 resolution. Figure 13 shows the timing comparison between the reference high quality solution (HQ) and the different conditions generated using the importance map. Calculations were done on an Intel Xeon processor running at 2.4 GHz with 3 GB of memory under Linux. Rendering the entire frame to the same detail as the task objects in Sub-gure 12(a) takes 5.2 times longer than the optimised method using only the saliency map, 3.5 with the task map and 4.5 longer than when using the task map and saliency map combined. Sundstedt et al. then went on to show, through a series of psychophysical experiments, that when performing a task in the environment (checking the re safety equipment), the viewers were unable to notice a difference between the high quality rendered animation, and the selective quality animation, which was rendered at a fraction of the computational cost [SDL 05].

4.6

Perceptual Realism in Real-Time

As we have seen, visual perception, including saliency and inattentional blindness can in fact be exploited to signicantly reduce the rendered quality of a large portion of a

22

(a) Task objects

(b) Task map IM(1, 0, +)

(c) Saliency IM(0, 1, +)

map

(d) Combined task and saliency map IM(0.5, 0.5, +)

Figure 12: Importance map examples from the corridor scene

Figure 13: Timing comparison for corridor A between the reference high quality solution (HQ) and the different conditions generated using the importance map

23

(a) Corridor A: The rst frame

(b) Corridor A: The 156th frame

(c) Corridor A: The nal frame

(d) Corridor B: The rst frame

(e) Corridor B: The 156th frame

(f) Corridor B: The nal frame

Figure 14: Corridors A and B from a 312 frame animation

24

Acronym HQ LQ TQ SQ IQ

Description High Quality: all pixels rendered using max settings (rp=16, st=0.01) Low Quality: all pixels rendered using min settings (rp=1, st=1) Task Quality:selectively rendered using only task map as input (IM(1,0,+) Saliency Quality:selectively rendered using a saliency map as input (IM(0,1,+) Task Quality:selectively rendered using saliency map and task map combined as input (IM(0.5,0.5,+)

Table 1: Animation pairs shown in the experiment: (a) HQ/HQ, (b) HQ/LQ, (c) LQ/HQ, (d) HQ/TQ, (e) TQ/HQ, (f) HQ/SQ, (g) SQ/HQ, (h) HQ/IQ, and (i) IQ/HQ. (rp = rays per pixel, st = specular threshold.) scene without having any affect on the viewers perception of the scene. This knowledge will enable us to prioritise the order and the quality to which any pixel in our image should be rendered. This does, of course, depend on the scene and knowledge of the task being performed. For many applications, for example games and simulators, such knowledge exists offering the real potential of using visual perception approaches to achieve perceptually realistic rendering in real-time.

Acknowledgements
We would like to gratefully acknowledge the contributions of Alans students in particular Veronica Sundstedt who has kindly provided much of the material on the selective guidance system and Importance Maps.

25

Figure 15: The UNC virtual pit experiment [MRWJ03]

Interaction in Distributed Environments

A particularly compelling example of a virtual environment, shown in Figure 15, is the University of North Carolinas Pit model [MIWJ02, MRWJ03]. Although the environment is rendered with relatively low graphical delity (by modern standards), a number of multi-sensory cues contribute to its overall richness. These include a telephone that rings, a radio playing music, an open window with a curtain blowing in the breeze, a carefully positioned fan to give the impression of that breeze, and passive haptics consisting of styrofoam blocks placed where virtual entities such as the walls of the room are located. Combined with a set of stressful tasks to perform, participants report a compellingly engaging experience. As the eld of virtual reality matures, our expectations of the tasks that can be carried out, and the behavioural delity of such environments becomes increasingly ambitious. Consequently, much recent research has focused on utilising additional sensory stimuli to create compelling environments [DWS 99, SSM 99, HMG03, TGD04]. Audio cues have been shown to increase participants sense of presence [PC99, LVK02] and studies have shown that in environments incorporating audio, subjects perform better in spatial localisation tasks than in the absence of this stimulus [ZF03, TGD04]. Haptic feedback has emerged as another important stimulus to exploit in many VR applications. More recently it has been incorporated in a number of single user and shared demonstrator applications [MBB 02, GHAH03]. Many are simple testbed tasks used to assess the value of collaborative haptic manipulation [BOH97,Hub02,OMJ 03, Rei04,KKT 04]. While they provide interesting results from a human factors point of view, it is difcult to transfer these to building shared applications designed to support haptic collaboration. In particular the impact of network issues of latency (the delay between the transmission of a message and its reception) and jitter (the variation in latency) commonly found on the Internet is not a solved problem. Of all the interaction modalities employed in multi-sensory environments, supporting haptic collaboration over wide area networks (WANs) is the most challenging due to performance requirements of the tightly coupled nature of the interaction. Therefore,

26

(a) Virtual biplane model

(b) Real biplane model

Figure 16: Virtual and real biplane models [AKH01] we will focus on potential pitfalls which arise in this case, and how to begin to mitigate them. A number of experimental studies (detailed later) have shown that haptic feedback is conducive to increasing a sense of presence (a feeling of being there) and copresence (a feeling of being together) between multiple participants working towards a common goal. Furthermore, the addition of haptic feedback as a means for input and constrained motion can lead to more user-centric interfaces [KW03]. However simply adding additional sensory cues such as haptic feedback to collaborative virtual environments does not, in itself, lead to compelling or usable applications. It is necessary to consider how the supported sensory stimuli can combine, together with rich behaviour, to reinforce the graphical and behavioural delity presented to the user. In training applications, a poor combination could potentially reinforce erroneous behaviour.

Effects of Haptic Feedback on Task-Performance

A number of studies, some of which are detailed in 6.1 and 6.2, have been carried out to assess the effects of haptic feedback on users ability to perform assembly, tapping and co-ordination tasks. A common limitation with these studies is that the results can be mixed, device or task dependent, and often difcult to directly map benets to real applications.

6.1

Single-User Case

Some single user studies have shown that the addition of haptic feedback improves task performance [GSW97, HGMR98]. In this section we will detail two in particular that are especially interesting: the rst is a virtual prototyping task carried out both in a haptically enabled training environment, and with a real model. The second more abstract study is a landmark molecular docking project. A single user experimental virtual prototyping task to assess the effect of haptics on a real training task was presented by Adams et al. [AKH01]. Three groups of partici-

27

pants were asked to attempt to build a Lego model of a biplane shown in Figure 16. One of the groups had no training, one trained using a VR simulator without haptics and one trained on the same simulator but with haptic feedback. The authors compared completion times for the task and found a statistically signicant change in performance attributable to the level of training received. In particular those subjects that trained in the environment with haptics out performed the others when building the real model. This experiment supports others that have shown skills gained in virtual environments to be transferable [RAB 00, TSRD01]. Early molecular docking research projects at The University of North Carolina, in particular GROPE, demonstrated that haptic display (when used to augment visual rendering) can improve users perception and understanding of forces and solid entities in virtual environments [BOYBK90]. Disappointingly, they found that typically only a two-fold improvement on task performance over a purely visual display was possible. However, this could be attributable to the limitations of available haptic devices at the time. Even now, current devices are not yet technologically advanced enough to provide both adequate tactile and kinesthetic feedback. Most importantly, however, Brooks et al. reported that the most valuable contribution from the additional haptic display to be in terms of users increased levels of understanding of the physics of drug docking sites. Although the direct benets of this may be difcult to quantify, for certain educational tasks this increased awareness would provide substantial benets. Now that we have outlined details of two important single user studies which analyse the effects of haptic display on task performance, it is appropriate to extend this to multi-user tasks. Studies performed in this context typically assess two factors: the rst being collaborative task performance, and the second being the level of co-presence experienced by the participants. We will rst detail a subset of those which analyse the basic core effects on shared task performance before describing a selection of experiments relating to co-presence.

6.2

Multi-User Case

Fitts law [Fit54] target acquisition tasks are increasingly being investigated with reference to haptic feedback [AW00,MGFB01]. One such experiment by Eva-Lotta Sallns et al. considers a task to hand over a cube to another user, together with tapping at target points within the context of Fitts law [SZ03]. To simulate plausible hand over rules, the model incorporated spring-like forces enabling both participants to feel forces exerted by each other. The hand over of randomly varying-sized cubes was assessed with and without haptic feedback. Prior to the actual assessed task users were trained to use the application with and without haptic feedback, until they felt condent enough to perform the task. This involved moving blocks down a series of shelves, handing over the block, and tapping specic shelves with it. The success of the users performing the task was measured in terms of time taken to perform error free trials, and the average time taken to hand over the blocks. No statistically signicant difference was found between the haptic and non-haptic hand over completion times. However a difference was found in terms of the average time taken to hand over the cube, and the size of the cube leading Sallns et al. to conclude that Fitts law holds for this type of task. They suggest that the lack of tangible 28

benet from the addition of haptic feedback may be due to the task only providing short haptic events as opposed to continued reinforcement. Another possible explanation given is that the spring-like force used to connect the user to the block provided a haptic buffer effect in the interaction, resulting in the recipient being able to move the cube before the rst user had relinquished control. One interesting result however, was that test subjects dropped more cubes in the non-haptic version than the haptically enabled version indicating that haptic reinforcement may be valuable in facilitating temporal spatial co-ordination during the exchange. Another example of a collaborative haptic tapping task was conducted by Margaret McLaughlin et al. This study was interesting because the task involved touch between participants. The experiment used a PHANTOM and a CyberGrasp device, with PHANTOM users being instructed to tap a model of the CyberGrasp users hand. A tactile Morse code was used which mapped the number of times a particular digit on the hand was tapped, to a letter in the alphabet. The participant receiving the tactile code recorded the letters they believed they had been sent. Results from this study showed that participants found it difcult to determine the codes sent to them haptically. However in most cases this was due to an overestimate of the number of times a particular digit had been tapped, rather than error determining which digit had been tapped. Interestingly a comparison of the most successful and least successful pairs of participants, revealed a potential but tentative correlation between success rate and feeling of co-presence. Each of these studies indicate very little real benet from the addition of haptic feedback to collaborative tapping task performance, and are contradictory to the single user examples described earlier. An interesting question to ask is why this might be the case? A number of possible explanations can be conceived, ranging from the choice of task to the choice of device. However in both single user experiments outlined above, the tasks being performed were continuous, intuitive and easy to relate to real world tasks. It is less clear if this is true for the Fitts law tasks described. Haptic interaction should ideally reinforce the users perception and graphical interpretation of what is happening in the environment, for sighted participants tasks which rely on haptic feedback to dominate over visual display are unlikely to show signicant positive benets. Furthermore, both participants perception of the environment, and task must correspond. If this is not the case, then the collaboration is likely to break down. Non-intuitive interaction would most likely exacerbate this.

Collaborative Experimental Studies

Specic studies involving users collaborating visually and haptically with continuous haptic feedback augmenting the visual display, are increasingly being conducted. These are especially interesting because they report on the sense of co-presence attributable to the addition of haptic feedback. Many such experiments involve two participants performing a simple carrying task in which each participant can feel forces exerted on the entity being shared. The following set of experiments described are particularly biased towards assessing co-presence, and are perhaps more like the prototyping and docking tasks in the 29

Figure 17: Basdogan et al.s wire ring experiment [BHSS00] sense that the interaction and both users perception of the environment is more in tune with their expectations.

7.1

Basdogan et al.s Wire Ring Experiment

The wire ring experiment by Cagatay Basdogan et al. is an important study on collaborative haptic manipulation [BHSS00]. The experiment itself required two users (one expert and one novice) to jointly steer a ring along a wire (shown in Figure 17), while minimising collisions. Only visual feedback was provided upon collision between the ring and the wire. Haptic feedback was employed purely to facilitate the communication between users manipulating the ring. The experiment was carried out rst with only visual feedback, then repeated approximately ten days later with haptic feedback. Two metrics were employed to assess the success of the task; a performance value and co-presence value (based on presence questionnaires [SW97, SSU00]). Mean and standard deviation values for their performance scores show a considerable improvement in task performance attributable to the addition of haptic feedback. Further, they show that the order in which participants carried out the experiments seems to matter. Groups which rst used the visuals only version followed by the visual and haptic display outperformed those which performed the tasks in the reverse order. The initial visual only experience appears to provide a training effect for the augmented haptic environment. In terms of co-presence, the study reported that in almost all cases participants reported an increased sense of being together in the haptically enabled environment. Further they found that factors such as age, computer usage, and gender differences seem to contribute to the level of co-presence reported. Interestingly they also report that users perception of the gender of the expert user differed between the haptic and non-haptic task. A signicant number of participants believed the expert user they were collaborating with in the haptic environment was male.

7.2

Hubbolds Collaborative Stretcher Carrying

Hubbolds stretcher carrying experiment is unique in the sense that unlike other similar experiments, the task is performed within the broader context of a complex virtual environment [Hub02]. The task itself requires two users to collaboratively carry a stretcher through a model of a process-plant shown in Figure 18. The simulation includes full 30

Figure 18: Hubbolds stretcher carrying experiment [Hub02] collision detection and a force eld based navigation technique to enable routing the stretcher, and three avatars two carrying the stretcher, and the patient through the model. A comparison between participants actually carrying a stretcher loaded with sandbags, and the virtual environment simulation was also undertaken. Qualitative ndings reported from the study included the observation that the virtual environment provides a good predictor for situations in which manoeuvring the stretcher is difcult. This nding was conrmed with the real stretcher carrying task. A second and compelling nding was that the addition of haptic feedback simplied routing the stretcher through the model considerably over the non-haptic version.

7.3

Sallns et al.s Cube Lifting Task

An interesting factor in the analysis of Sallns et al.s cube lifting task [SRGS00] is the use of video taped recordings of participants performing the task together with presence questionnaires allowing a detailed post experimental analysis of task performance. Two out of ve tasks were analysed, each requiring two participants (represented by spheres) to jointly lift cubes. The rst involved both participants lifting eight cubes (shown in Figure 19) in a specic sequence with the aim to build one cube, without a visual illustration of the target solution. The second task required the participants to collaborate to build two piles of cubes in a specic sequence from the eight cubes in the environment. For this task both participants were provided with a visual illustration of the target solution. Participants carried out the tasks both with and without haptic feedback, using the haptic device purely for input in the non-haptic case. A two minute training session was employed to familiarise participants with the features of the environment before they undertook the specic tasks. Reported ndings from the experiment were somewhat mixed, task performance was shown to improve with the addition of haptic feedback but no signicant increase in co-presence was reported. Interestingly however, there appeared to be a statistically signicant difference between the participants ability to lift the blocks depending on whether the target solution was supplied, implying (as one would expect) that a prior knowledge of the objective facilitates collaboration.

31

Figure 19: Sallns et al.s cube lifting task [SRGS00]

Main Challenges

The benets offered by haptic interaction in shared virtual environments is not exactly clear cut. This is due to any number of possible factors ranging from choosing the correct metaphors to facilitate human computer interaction to device limitations. However, in the right application context and with appropriate interaction metaphors, a signicant benet can be achieved as evidenced by the GROPE project [BOYBK90]. When developing collaborative haptic applications, it is essential to ask the following questions: What types of interaction will the application need to support? Will the interaction feel intuitive? Will employing haptic feedback greatly benet my application? Each of these questions require careful consideration as the answers will impact signicantly on the usability and success of the application. A signicant challenge is the design of interaction metaphors for haptic communication. For example, does the user manipulate an entities position directly or indirectly? This is very much an ongoing research problem, especially as it is heavily application and device dependent. Further investigation is needed to determine the types of interaction metaphors that users prefer when undertaking classes of haptic tasks. We do not propose to solve the problem here, rather in 9 we present the types of interaction modes possible in shared haptic virtual environments and the application contexts within which these can be employed. Two more major challenges are briey outlined in the following subsections: those of shared manipulation of the same entity and distribution over the Internet.

8.1

Shared Manipulation

Two or more participants manipulating the same entity poses a major problem for collaborative haptic environments. There are two signicant and related barriers to achieving such closely coupled interaction. The rst is to do with the choice of simulation 32

methods, and the second is synchronisation of state information across participants involved in the collaboration. Robust simulation is required in situations where it is critical that the position computed for the shared entity is constrained and absolute, as is the case for articulated rigid bodies. A solver needs to be employed in order to solve complex constraints on motion, and this inevitably increases the complexity of the simulation. Many of the simple collaborative haptic demonstrators tend to employ fairly basic force models often using spring-like forces to indirectly manipulate shared entities rather than absolute positional constraints. One of the reasons for this choice is that this provides a built in tolerance in the actual position and orientation that the entity may occupy. For many applications this may sufce but for complex environments and critical task training, more sophisticated simulations and force models may be more appropriate. Related to the issue of simulations is the problem of ensuring that each participant sharing an entity has a consistent view of the environment. The consequences of differing state could be catastrophic as each users perception of the interaction being performed would differ thus leading to a breakdown in the collaboration. Exchanging state information between participants on a local area network (LAN) is generally fast and reliable, however a wide area network (WAN) such as the Internet brings with it some major challenges detailed later.

8.2

Distribution over the Internet

An interesting point to note regarding all of the collaborative haptic studies described earlier is that none of them operate over a real WAN such as the Internet. In practise there are few studies in which this has been attempted, an example of one such study is the collaborative stretcher carrying example by Manuel Oliveira et al. which illustrates the potential difculties that arise [OMJ 03]. The task required two participants, one at University College, London and the other at The University of North Carolina to collaboratively carry a stretcher using force feedback devices. In preliminary trials using the Dive [Hag96] system, the authors concluded that the task was not possible due to rapid divergence (caused by network latency) between the haptic and graphical states. Consequently a bespoke application was built to attempt the task, however the authors concluded with a need for a better physical model and strategies to synchronise simulations as well as mitigate the effects of network latency. Techniques for synchronising state across multiple participants on a WAN are an important consideration for collaborative virtual environments and depend primarily on distribution architecture choices. For behaviour-rich multi-sensory environments these issues become even more crucial as potentially complex state information needs to be kept consistent. In order to better understand how participants may collaborate in haptically enabled virtual environments, it is useful to consider the types of interactions that applications may be required to support. These interactions, many of which may modify the state of the environment, also need to be distributed across multiple participants.

33

Figure 20: User colliding with an entity

Classes of Interaction in Shared Virtual Environments

Broadly, the types of interactions possible in shared virtual environments are either collisions, touch, or more complex actions which have causal relationships with the user or other entities in the environment. In the following subsections we break down this broad categorisation further, in order to detail the types of complex interactions possible.

9.1

User-Entity

User-entity interactions are those which occur between a user and any specic entity in the environment. Figure 20 shows a user (represented by a small shiny black sphere) colliding with a torus. This collision may not provide a haptic response thus it is considered separately to the case of a user touching an entity shown in Figure 21. Continuously touching and exploring the surface of an entity is an altogether different user-entity interaction relationship as the users actions and haptic response are closely coupled. A third user-entity interaction relationship is that where the user moves the entity as shown by a sequence in Figure 22 (muted versions of the users sphere representation and the torus are historical positions and orientations). Haptically moving a entity may take one of two forms, direct manipulation by selecting it and dragging/dropping it or indirect manipulation by exerting a force on it. Direct manipulation is more akin to lifting and moving objects in the real world however in haptic environments the latter is more commonly employed.

9.2

User-User

In haptic environments, possible user-user interactions could typically take the form where a user collides, or touches another user, as shown in Figure 23. Collisions between users, with or without, haptic response may have consequences such as encouraging the user that has been collided with to move out of the way, or both users to back off. Furthermore, in a haptic environment users do not need to collide to touch, potentially they may interact via indirect forces (illustrated as a curved line between both users). 34

Figure 21: User touching an entity

Figure 22: User moving an entity A more common user-user interaction typically found in virtual environments is the ability to converse as shown in Figure 24. In the past, applications have adopted anything from text based communication to Voice Over IP.

9.3

User-Entity-User

Interactions in the category user-entity-user can be fairly sophisticated. Firstly, a user colliding with an entity can cause it to move and in turn collide with another user as shown in Figure 25. Although this interaction can be seen as two user-entity interactions, it is described here due to the causal relationship of the interaction. Two or more users may also simultaneously touch the same entity as shown in Figure 26. This situation is not too complex as long as each user does not invade the space of other participants. In the case where users try to ght over a particular region, the simulation becomes complex as each users representation potentially collides and should provide a proper haptic response in addition to that between the users and the entity. The interaction scenario shown in Figure 27 where two or more participants ma-

Figure 23: User touching or colliding with another user

35

Figure 24: User talking with another user

Figure 25: User colliding with entity which in turn collides with another user nipulate the same entity is possibly the most complex interaction situation possible. To achieve this requires the state to be synchronised between participants in addition to collision detection and haptic response between user representations, and between users and entities. This is the worst case scenario both from a simulation and distribution perspective, especially if the entity being manipulated is complex and changing its state may have a knock-on effect elsewhere.

9.4

Entity-Entity

Entity-entity interactions of the type shown in Figure 28 of a torus colliding with a cone are fairly straightforward but may also have consequences depending on the physics of the environment. For example, upon collision the torus may come to a standstill, it and the cone may move off together, or it may cause the cone to move off in a

Figure 26: Two users touching the same entity

36

Figure 27: Two users moving the same entity

Figure 28: An entity colliding with another entity different direction to its continued motion. Haptically however this type of interaction, depending on the complexity of each entity involved, presents a signicant collision computation problem within haptic performance demands. An entity exerting a force on another entity, as shown by the curved line between the torus and cone in Figure 29 may exhibit complex behaviour depending on the nature of the force between the entities. Further complex interactions can be achieved through combinations of each of the above described interactions. Consequently, these types of interactions need to be appropriately synchronised between all participants if a meaningful collaboration is to be undertaken.

Figure 29: An entity exerting a force on another entity

37

10

Common Shared Interaction Modes

Considering the specic case of shared interaction with the same entity, a number of interaction modes may be employed to manage the collaboration. The mode employed should be dependent on the demands of the application. Shared haptic applications need not support simultaneous manipulation of the same entity unless it is specically required. In the following subsections we briey describe three possible shared interaction modes and their practical usefulness in a collaborative application.

10.1

Turn Taking

In a turn taking (or collaborative [BOH97]) interaction mode, each participant collaborates by performing their action in turn. While a user is manipulating the entity, others watch and await their turn. In many prototyping, training and gaming applications this mode of manipulation is both natural and intuitive. The entity being manipulated is commonly locked, using a token, to a specic participant for the duration of their interaction. When the token is relinquished the entity is available to another participant.

10.2

Free-for-All

The free-for-all (or co-operative [BOH97]) interaction mode is tightly coupled with participants simultaneously manipulating an entity. It is commonly found in the real world when people need to move or manoeuvre a bulky heavy object. In virtual environments it has also been employed for similar tasks however, in a training application it may provide an excellent method for participants to interrupt a specic sequence of actions. For example, user Rob is demonstrating an assembly maintenance operation to a group of users but Raf has some difculty understanding the sequence; he may choose to take hold of the entity Rob is manipulating, in order to intervene, and pause the sequence.

10.3

Guided Hand

Guided hand is another tightly coupled interaction mode however, it differs from the free for all mode in the sense that there is always one active and one (or more) passive participants. The active or dominant participant may be an expert user training other participants how to perform a complex task. This type of interaction is most likely to be suited to surgical training applications [GHSA05].

11

Distributing Simulations

We have already stated that when distributing simulations over a network, each participant must experience a coherent view of the simulation. This is even more critical in applications in which users may interact with and affect the outcome of the simulation. It would be difcult for multiple participants to undertake a meaningful collaborative task involving interacting with the same rigid or deformable entity, if each person did

38

not see a consistent and plausibly correct simulation of its behaviour. Some state information relating to the simulation (such as positional updates, forces, constraints, changes in size etc.) needs to be sent over a network to other participants. With any network there is some elapsed time (latency) introduced between sending a message and that message being received.

11.1

Injection Problems

Figure 30 shows a number of different simulation scenarios involving two participants each running their own instance of a simulation, over different network conditions, with events from each being injected into the others simulation [MGPH05]. The rst simulation 30(a) occurs over a low xed latency connection, events from each participant arrive in time to be taken account of in the simulations and both remain consistent with each other. In the second case 30(b), the simulation occurs over a high xed latency connection. Each participants events arrive consistently late, and so the order in which these are incorporated into the simulations differs but in a predictable manner. The third case is more complex and represents a variable latency connection 30(c), notice how the order in which events are introduced into the simulations is much more haphazard and difcult to predict. Finally the most catastrophic case 30(d), in which the latency is variable and events may even be lost, results in very different outcomes for both participants. Depending on the network protocol used, this loss of updates may be impossible to detect. TCP/IP guarantees to deliver updates by transparently re-sending lost updates incurring a signicant additional latency. UDP on the other hand attempts to deliver the updates in the order in which they arrive, but offers no notication of lost updates. This can have serious consequences on simulations particularly if the lost update makes a signicant state change [MGPH05].

11.2

Users Interaction Information

Depending on the application and the possible tasks that it allows users to perform, a number of different interaction related state updates may need to be distributed. Shirmohammadi et al [SG01] introduce the concept of interaction streams, in conjunction with reliable transfer for key updates, in order to share interaction data to facilitate collaboration. This approach recognises the fact that different applications impose different demands on the reliability and update rate of shared interaction state. Using the concept of interaction streams high priority interactions (such as user-entity-user, or entity-entity) which have a causal effect, could be sent using a different network protocol compared to lower priority ones. In a simple walk-through, incorporating simulations of ocking birds which the user does not directly affect, only the users positional updates need to be distributed. Since the purpose of simulation in this case is purely to provide some contextual dynamic behaviour in the environment, each participants simulations may be computed locally thereby reducing the distribution requirements. If however users are able to direct and change the path taken by the ocking birds, then this change must be distributed to other participants. This can be achieved by either sending positions of all the ocking entities or by sending just the events that caused the change, which is more 39

(a) Constant Low Latency

(b) Constant High Latency

(c) Latency and Jitter

(d) Latency, Jitter and Packet Loss

Figure 30: The effect of different network conditions on distributed simulations [MGPH05]

40

efcient but may result in subtle differences in each independent simulation. For a purely gaming application, these subtle differences in state may not be critical but this is not always true for all simulations in all applications. Consider a collaborative CAD application employing deformable object simulations. If two or more participants are required to co-operatively lift and position a deformable component then each users position and interaction information must be distributed. The question is, how best to distribute the state changes for the shared entity. The answer depends both on the network conditions and the techniques employed to compute the deformations. If an iterative numerical method is employed, stability and convergence [But87] problems can arise between separate simulations, particularly when events are delayed and injected late or lost [BT89]. Consequently participants simulations will rapidly accumulate errors, and diverge making any collaboration impossible.

11.3

Synchronising State

The problem of divergent simulations is solved by synchronising state information across all participants. This is essentially achieved by maintaining a single locus of control [Pet99] for complex simulations. A single server or simulation manager maintains a consistent simulation state, and determines how this is synchronised across all participants. Exactly how this is mapped to distribution policy and architecture, depends on the type of network that the application uses [BT89,Dah99,Fuj99]. There are a variety of different ways in which this can in fact be congured, while still maintaining a logical single locus of control. Real networks are very different and the amount of xed, variable latency, and update loss differs signicantly between LAN and various WAN networks. To better understand how to implement suitable distribution policies and which architecture to adopt, it is necessary to understand the network conditions under which they will be employed.

12

Real Networks

Real wide area network characteristics vary widely but can be typically characterised in terms of two metrics, the amount of latency and jitter. Latency is the delay between a message being transmitted and received and is unavoidable, as no matter how fast networks become they are fundamentally limited by the time taken to transmit data over a given distance through the carrying medium. We normally encounter latency in long distance telephone conversations, when faced with such communications delay, in structured conversations people may adopt a wait and talk (turn-taking) strategy or more often curtail the conversation [EM63, RK63]. Latency can increase signicantly if a network is heavily loaded as the sender must wait until the network is free of trafc (in the case of Ethernet) before it can send a new message [CDK01]. Jitter; the second metric, is the actual variation in latency, often this can be attributed to packets being lost and needing to be re-sent. While it can be argued that with better network quality of service jitter will cease to be a problem, due to nancial cost of improved network infrastructure it is still likely to persist for the foreseeable future.

41

Destination Ofce LAN (Switched Ethernet) The University of Bristol, UK FCS, Amsterdam, The Netherlands Labein, Bilbao, Spain (night) University of North Carolina, USA Labein, Bilbao, Spain (afternoon)

Distance (km) 0.01 230 496 1140 6062 1140

Lost Packets 0 0 0 3 0 68

Round-Trip Time (ms) Min. Max. Mean Mean Deviation 0.36 13.9 20.9 62.3 104 87 13.2 88.2 164 663 106 2298 0.46 14.3 27.6 68.3 104 341 0.047 0.34 4.54 4.59 0.038 136

Mean Round-Trip Update Rate (Hz) 2170 69.9 36.2 14.6 9.62 2.93

Typical Hop Count 1 14 18 17 18 17

Table 2: Results of sending 8000 packets to various destinations from The University of Manchester, UK [MGPH05]

12.1

Network Latency and Jitter

The aim of any distribution policies and architecture facilitating collaboration, is to present a consistent and coherent view of a shared environment to each participant [MGPH05]. Ultimately the speed of light imposes a nite upper bound on how fast information can be communicated across a network, and in practise real network performance is well below this level. Consequently, regardless of architecture, two remote users cannot share an instantaneous, fully-synchronised view of an environment [Mar02]. Faced with this challenge the majority of collaborative virtual environments adopt one of the two most common network distribution architectures, client-server or peer-topeer [SZ99]. Both these architectures have their own benets and shortcomings detailed later. Table 2 and Figures 31 to 36 illustrate the result of measuring latency and jitter on a number of network routes between The University of Manchester and our research partners. These routes differ in performance by as much as one or two orders of magnitude. Consequently architectural choices appropriate for a LAN are not necessarily going to apply for typical WANs such as those proled, and likewise applications designed specically to work over WANs are unlikely to take full advantage of the faster local environment [MGPH05].

12.2

Impact on Task Performance

A number of studies have examined the effectiveness of collaborative virtual environments and groupware applications. In particular, these have considered factors that affect the extent to which such environments can be successful, including network latency, collaboration strategies, rendering delity, and interaction mechanisms. Participants have been shown to adapt to performing tasks in the face of xed network latencies up-to 200ms [PK99]. However, the amount of xed latency tolerable is highly dependent of the task and collaboration strategy adopted. Often in situations where users are able to tolerate high latency, it is attributed to a move and wait strategy being adopted. Jitter potentially has a much greater impact on the users ability to coordinate their actions, particularly with regards to predicting the actions others [Gut01]. Changes in participants natural interaction strategies to compensate for the effects of latency and jitter can reinforce erroneous behaviour/responses, demonstrated in stud42

Figure 31: Measured performance on The University of Manchester local LAN [MGPH05]

Figure 32: Measured performance to The University of Bristol, United Kingdom [MGPH05]

Figure 33: Measured performance to FCS, Amsterdam, The Netherlands [MGPH05]

43

Figure 34: Measured performance to The University of North Carolina, USA [MGPH05]

Figure 35: Measured performance to Labein, Bilbao, Spain (afternoon) [MGPH05]

Figure 36: Measured performance to Labein, Bilbao, Spain (night) [MGPH05]

44

ies with teleconferencing applications [RJ99]. In the context of virtual training applications, if a network glitch causes a sudden and surprising discontinuity in motion, the trainee may believe that they caused it by making an error, even though they were performing the correct operation. For graphical display of 3D environments, one of the most important considerations is maintaining suitably high frame rates. This is considered to be approximately 20 30Hz [Bro99]. Below this level motion appears discontinuous, and entity interactions such as collisions may not be correctly rendered due to the high inter-frame latency.

12.3

Architectures for Distributed Systems

The two main architectures adopted for distributed systems are client-server and peerto-peer, each of these architectures have their strengths and weaknesses and the choice of which to adopt depends on the demands of the application. At the heart of any distribution architecture is a choice regarding where control should reside, and for many applications, this decision is not always intuitive. Therefore, in addition to the basic distribution architectures a number of hybrid approaches have also been proposed. The relative merits of a number of approaches are discussed in the following sections. 12.3.1 Client-Server

The client-server architecture (shown in Figure 37) is one of the most straightforward architectures to use for supporting rich behaviour. It uses a centralised server that runs a simulation of the virtual environment and communicates relevant state changes to all the clients. Managing state across all clients is simple and performed by the server. Clients contain a representation of the application for graphical rendering, but do not perform any simulation activities locally. User interactions are sent directly to the server for processing and clients only update their local representations when instructed to do so [Mar02]. Instead of distributing events which need to be injected into separate simulations, information describing the outcome of a single centralised simulation is communicated to remote clients. This architecture is robust against the effects of jitter and latency, as when update events suddenly experience an increased delay the graphical effect will be no more serious than a temporary disruption of a television programme. The client may briey display an inconsistent view of the world, but this is quickly corrected when the update does eventually arrive. In addition to the synchronisation benets, with a single point of contact, application start-up is simplied, and it is straightforward to have the environment persist even when all the clients disconnect [Mar02, MGP 04, MGPH05]. The biggest disadvantage of the client-server approach is that the clients local view of the environment is only updated after a round-trip to the server, resulting in additional input latency for all connected participants. However it is generally accepted that this architecture is the most appropriate to support co-operative tasks in collaborative virtual environments due to the need for consistent state [BOH97]. When latency is too high (as is the case between Manchester and Labein) the graphical update rate requirement of at least 2030Hz cannot be easily met using a clientserver approach. Consequently the users input is seen to lag and their perceptions 45

Figure 37: Client-server architecture [MGP 04] of causal relationships begins to break down [Mic63]. The effect this actually has is likely to be task dependent, and experimental studies disagree with regard to the threshold at which task performance is affected. Some reports suggest that participants can tolerate constant latencies of up-to 200ms without a signicant degradation in task performance [PK99], while others suggest the gure is much lower [MW93]. Our earlier analysis of the round-trip latencies on our ofce LAN, and the Internet connections between The Universities of Manchester and Bristol, and between The University of Manchester and FCS-Control Systems in Amsterdam indicate that network conditions are sufciently fast for a pure client-server architecture to be adequate (assuming no signicant latency is added by simulation processing on the server). Occasional peaks of latency higher than 50ms, are likely to cause brief perceptible glitches in continuous movement [MGPH05]. 12.3.2 Peer-to-Peer

A peer-to-peer architecture on the other hand has each computer (or peer) directly applying a users input locally to their own instance of a simulation, while simultaneously communicating these events to other peers as shown in Figure 38. The main benet of this approach is that it avoids the additional input latency present in the client-server approach, and hence has proved a popular architecture for a number of collaborative systems [MZP 94, BBFG94, GB95, Hag96]. For many applications with low synchronisation requirements (such as multi-player war games, viewing scientic datasets or static models, simple carrying tasks, and primarily social environments) such architectures are successful, particularly over low latency low-jitter connections [MGPH05]. However, in complex behaviour-rich environments synchronisation issues begin to dominate. In the absence of a server to arbitrate, when simulation states computed by each peer diverge catastrophically (due to updates either being discarded, arriving in different orders, or at different times at each peer), there is no obvious way to correct the causal inconsistencies that arise [MGPH05]. Peer-to-peer architectures are most successfully employed on networks with an update delivery rate similar to that shown in Figure 30(a). Events are delivered in the same order to all peers and with a suitable time-stamping and simulation roll-back scheme, updates can be applied at the same time to all simulations. Depending on the network protocol used, however, packet loss can still pose a signicant synchronisation problem [MGPH05].

46

Figure 38: Peer-to-peer architecture [MGP 04]

Figure 39: Enforced turn-taking (locking) architecture [MGP 04] 12.3.3 Shades In-Between

Due to the limitations of the two main distribution architectures detailed above, a need exists for hybrid approaches. A third conguration commonly used attempts to combine the low-latency advantages of a peer-to-peer network, with the synchronisation benets of a client-server architecture. This is achieved through a peer-to-peer system which enforces object-based locking as illustrated in 39 (an architecture occasionally called token-ring [BOH97]). This eliminates the problem of out-of-order updates as only one user is able to interact with a given simulation object at any instant in time, unfortunately however it also prevents simultaneous co-operation. Again, for many applications this architecture will be sufcient, but it imposes strict turn taking which may be unintuitive in certain application contexts. For example in many interactions, such as holding a conversation, social conventions naturally result in turn-taking, but participants still expect to be able to interrupt others [MGPH05]. An improvement to this approach is to provide the ability to identify and lock specic regions of a peer-to-peer network into work groups, in which all related peers share network characteristics similar to those shown in Figure 30(a). An architecture similar to this was suggested by Hespanha et al. [HMS 00]. One problem with architectures such as this, though, is that it is not always clear to the participants who is allowed to interact at any particular instant in time. Neither is it clear how best to assign participants to specic work groups. This can result in the situation where users on the periphery of a fast network group are able to collaborate with all other participants except each other, when either one joins the interaction they lock out the other, while the rest of the group appears free to interact at will [MGPH05]. An alternative architecture proposed by Marsh et al. [MGP 04, MGPH05] utilises a number of distributed simulation servers, typically with one per local network as

47

Figure 40: Roaming server architecture [MGP 04] shown in Figure 40. One of these servers is nominated to manage the environment itself, ensuring that logically only one of the simulation engines can be active at any time. When the system starts up, all of the simulation engines are dormant. Clients connecting prole the network link to each available simulation server, in order to determine their preferred one for subsequent interactions. When a user begins to manipulate an entity, their client requests that its preferred server is activated. If all the simulation engines are dormant, this request is granted, and the client notied. All updates now occur between the client and the active fast local simulation server, but in an identical manner to a standard client-server architecture. If a second user now joins the interaction to either collaborate or co-operate with the rst, their client sends a request to activate their preferred server. In this case, the request is declined and the requesting client is informed to use the existing active simulation server. Since both users interact using the same server they are free to interact with the same simulation entities. However the second user may suffer greater latency than the rst depending on their geographic location and network connection. A list of actively interacting users is maintained and if a period of two seconds elapses in which no-one holds any entities, the simulation servers synchronise and enter their dormant state. The system is now ready to activate whichever simulation engine the next user to initiate an interaction requests. If during this two second interval a participant selects an entity then the synchronisation process is cancelled in order to maintain predictable behaviour.

13

Considerations for Behaviourally-Rich Distributed Haptic Environments

We have mentioned the required update rate for graphical rendering, however for haptic rendering a far greater demand exists. It is widely believed that an update rate of at least 1KHz [Bur96] is necessary for correct perception of solid contact; below this rate entities begin to feel spongy, and if the rate drops too low, instabilities will arise. Given the types of networks proled, it is a signicant challenge to maintain a fast enough update rate for co-operative manipulation of a shared entity. In order to see

48

how this may be supported and the restrictions which apply, it is useful to understand some of the strategies adopted to mitigate the effects of latency and jitter.

13.1

Techniques for Managing Latency and Jitter

A number of techniques have been successfully employed to manage limited latency and jitter for specic applications or tasks. These include using a peer-to-peer approach, time warping [Jef85,MVHE04,Mir00], additional local latency [Mau00], buffering/replay [MPW99], and predictive/dead reckoning [BHM 92, MZP 94, WZ98, WKSKH99] techniques. While peer-to-peer architectures save on the round-trip requirement of client-server approaches, we have seen that synchronisation issues rapidly begin to cause problems in peer-to-peer systems with heavy simulation synchronisation needs. Thus for behaviour-rich environments, this architecture is only really suitable to employ on fast networks. Many existing peer-to-peer applications work due to the relative simplicity of the behaviour supported. Computation of costly multi-body simulations is often parallelised. However synchronisation issues present a signicant challenge, even to shared memory supercomputers, where communications latency is many orders of magnitude less than that on a LAN. Parallel discrete-event simulation relies on concepts such as virtual time to enabling time-warping, allowing simulations to be rolled back to a consistent state when delayed updates cause conicts between state. The use of this concept has been suggested for virtual environments [Jef85,MVHE04,Mir00], however its lack of widescale adoption stems from the need to maintain a history of simulation state, and the ability to rapidly move forwards through time in order to return to the current time. This either implies signicant spare CPU capacity or is likely to have the counter effect of introducing further latency [MGPH05]. The concept of introducing an additional delay to the application of locally generated events additional local lag [Mau00, MVHE04] has been suggested in order to resynchronise updates such that they are incorporated as in Figure 30(a). This requires all input events, whether locally generated or remote, to be delayed at each peer to the latency of the slowest network link. While this solution might be reasonable to adopt under the conditions similar to the connection between Manchester and The University of North Carolina (where there is low jitter and constant latency), to mitigate the effects of jitter, all updates must be delayed by the maximum latency of the slowest network connection available. For many of the networks shown in Table 2, this threshold is much greater than the mean round-trip time. The ManchesterAmsterdam connection requires that total delay should be consistently 82ms, whereas the mean client-server round-trip would only be 27.6ms. Similarly for the Manchester-Bristol connection a client-server architecture would experience a 14.3ms mean round-trip, but to compensate for even the relatively low level of jitter requires all updates to be slowed to at least 44.1ms when using an additional local lag approach [MGPH05]. The converse solution of delaying upon arrival and buffering updates by the duration of the mean latency plus the mean jitter, which are then interpolated using cubic curves is particularly suitable for applications in which users naturally adopt a turn

49

taking strategy. On slow or jittery network connections, this approach allows passive observers to view smooth, accurate playback of motion sequences [MPW99]. Predictive or dead-reckoning techniques rely on a local representation of entities to continue their motion, based an extrapolation of historical information, in the absence of updates from the server [BHM 92, MZP 94, WZ98, WKSKH99]. If an update arrives which diverges from the dead-reckoned state of an entity, a brief glitch may be perceived. This solution works well for emergent behaviours involving large numbers of autonomous entities, where such brief glitches are relatively unimportant. Wilson et al [WKSKH99] present a solution to mitigate the effects of latency in networked haptic applications which uses two approaches. The rst models the variablelatency communication link as a passive electricity transmission line with varying mass and stiffness (virtual coupling). This creates a system with variable performance but guaranteed stability for the haptic device. The second technique employs haptic dead reckoning, keeping a complete world model on a server and a local simplied model on clients. This allows the clients to haptically display models at the 1KHz update rate, while receiving occasional corrections from the more accurate server model. A similar solution is employed by Marsh et als haptic demon which maintains a local haptic update rate in the absence of updates from the server [MGPH05].

13.2

Client-Server vs Peer-to-Peer

For the specic networks proled, it is interesting to see how client-server and peer-topeer approaches compare. For low latency and low jitter networks like the ofce LAN both solutions perform very well for applications with high synchronisation demands. The client-server approach is easier to implement from the point of view of provision of synchronisation strategies but ultimately both architectures perform equally well. In the case of the low latency but high jitter networks, the client-server architecture is more appropriate than peer-to-peer, as the additional cost of employing techniques such as additional local latency to mitigate the effects of jitter is prohibitive. For high constant-latency networks, the peer-to-peer architecture offers a good solution to mitigating latency by minimising the round-trip time, especially when used in conjunction with predictive techniques for hiding latency. However, for high latency and high jitter networks a hybrid architecture is necessary as neither client-server not peer-to-peer solutions will facilitate meaningful co-operative collaboration. In a turn taking application however, the above mentioned buffering techniques may be employed to present smooth motion to participants using a client-server approach.

13.3

Rich Behaviour vs Haptic Response

Recall that the graphical rendering update rate is required to be in the region of 20 30Hz, while for haptic display it must be 1KHz or better. This fact means that environments employing simulation with purely graphical display can afford more delay than those employing haptic display. Provided the latency introduced by simulation computation is not too high, the required update rate can be met on a LAN [MGPH05]. Although it is much more challenging to mask the latency for haptic rendering over a

50

WAN than it is to do so for the simulation case alone, recent evidence suggests that the amount of haptic latency tolerable is task and device dependent. Ferrell [Fer66] performed experiments involving two tasks; a contact task and a spring loading task. The study introduced latencies ranging from 0 1 second into the two tasks, and participants were trained to use either a continuous interaction or move and wait strategy. In the contact task, the move and wait interaction strategy was preferred while for the spring loading task the opposite was true. The author concluded that a move and wait approach was preferable when haptic response is indicative of a positional error, but less effective than continuous interaction in search tasks. Instabilities arising from latency were observed as bouncing/oscillations during sustained contact. Consequently the primary advantage of haptic feedback, a closely coupled interaction and response loop, is lost through the introduction of latency. Jay et al. [JH04] show that in target acquisition tasks in which haptic feedback is intermittent and potentially non-essential, latency in the supply of haptic feedback has no perceivable impact. Additionally while a visual latency of 75ms affected task performance, subjects were not aware of the degradation until it reached 90ms. In a subsequent study of a reciprocal tapping task in which haptic feedback was more critical to the task, Jay et al. [JH05] show that while visual latency started to impact upon task performance at around 70ms, haptic latency did not signicantly affect it until around 200ms. Subjects seemed to have difculty perceiving the haptic latency even when it did begin to affect task performance. In both these studies however haptic feedback is related to a discrete event, and so results may differ substantially for continuous manipulation. In a pilot study in which subjects maintained contact with a solid surface, Marsh et al [MGPH05] state that subjects reported surfaces as feeling strange with additional latencies of around 2530ms. Signicant instabilities became apparent at around 40ms. The impact of latency on haptic task performance however is the subject on ongoing research and still requires further study. The graph shown in Figure 41 shows the latency introduced by collision, geometric constraint [Lab99] and deformable object [Wei02a, Wei02b] computations carried out by the simulation server in a collaborative CAD prototyping application. This performance information was recorded while carrying out a shared assembly task using model of a blood-pressure monitor shown in Figure 42. The two areas of high computational cost are associated with docking complex components, which potentially match a large number of constraint axes. These regions introduce between 1017ms of computational latency, when this is added to the network latency the total latency becomes too high for stable haptic rendering over a range of the proled WANs.

14

A Brief Survey of the State of the Art of Collaborative Haptic Environments

The characteristics of the networks proled, together with the demands of simulation and haptic display, illustrate how challenging a goal shared co-operative haptic manipulation really is. It is nevertheless an active research area, with a number of groups undertaking studies to better understand the problems associated with achieving shared

51

Figure 41: Latency introduced by the simulation server [MGPH05]

Figure 42: The blood pressure monitor test case [MGPH05] haptic interaction over the Internet. Some of the solutions proposed are discussed in the following sections and are classied broadly according to distribution choices.

14.1

Peer-to-Peer

Manuel Oliveira et al. [OMJ 03] report on lessons learnt, when implementing a cooperative haptic application to manipulate a stretcher using the DIVE virtual reality framework [Hag96] (DIVE provides a peer-to-peer distribution approach with a replicated database [FS98]). Their haptic stretcher carrying task required participants to follow a specied path into a building and position a stretcher onto a target, and was carried out between The University of North Carolina (UNC) and The University College London (UCL). A participant at UNC acted as a confederate during the experiment aiding the UCL subjects to carry out the task. During the experiment the authors found system performance to be especially poor

52

making it impossible to complete the task. They attribute this to the different frequency of updates to parallel data-structures (haptic and graphical). A haptic scene graph was used to locally calculate forces to apply to the haptic device, but both the haptic scene graph and graphical scene graph had to be synchronised for the haptics to reinforce the visuals and visa versa. Since the haptic and graphical update rates are very different, this presented a problem as the event loop in DIVE is tightly coupled to the rendering loop allowing events to only be processed at the rate of graphical rendering. This coupled with network latency resulted in the haptic and graphical scene graphs diverging rapidly. Consequently actual co-operative experiments were performed without haptics, and employed visual and audio feedback however the lack of physical constraints in this case rendered the task less realistic resulting in a loss of presence and co-presence. Furthermore, the authors experienced other problems with shared manipulation, relating to inconsistencies in users positions during the simulation caused by a loss or late application of events. In order to address this, they sent several notications of each event discarding duplicates locally. The reported result of their study was that joint manipulation of the stretcher using their system posed a number of synchronisation issues. The choice to allow interaction with a local model distributing rotational and translational changes to other remote participants, can be assumed to be synchronised when events are applied in the same order by each client, and generated at a higher resolution than the frequency of the manipulation of the stretcher. If this event delivery characteristic is not met then the divergent states of each instance of the stretcher causes signicant jitter of its position. The simulation performed well in tests on the LAN at UCL but when trialled between UCL and UNC the simulation was difcult to synchronise. In order to solve this problem, the authors employed a script to update the local stretcher based on the state of a shared global representation. The stretcher was then aligned locally based on the position/orientation of its handles thus avoiding direct co-operative manipulation. In an alternative bespoke application two users co-operatively lifted a cube over a height threshold and maintained it there for a pre-determined time. An additional contextual cue (semi-transparent version of the box) was provided to indicate to each participant the state as perceived by the other. The multi-threaded application employed a peer-to-peer architecture communicating simple reaction forces injected locally into the simulation. Updates were transferred via UDP, with the authors concluding that the experiment required sophisticated buffering, a better physical model, synchronisation and smoothing. Further work based on this study assessed the levels of presence and co-presence achieved during the haptic collaboration [KKT 04].

14.2

Client-Sever

Pietro Buttolo et al. [BOH97] argue that for purely graphical display users adopt a move and wait strategy, to restore hand eye co-ordination. In the case of haptic display, delay in processing information can cause instabilities. Therefore the focus for shared haptic simulation is to reduce delay in processing force information. In order to study architectures for addressing this, the authors investigated three classes of applications:

53

 haptically enabled environments collaborative haptic environments co-operative haptic environments Within the context of these applications, they discuss a scenario involving two participants moving a heavy engine block across a oor, stating that a high quality simulation of this task would be impossible with the typical latencies found on the Internet. Two interaction mechanisms relating to haptic display are were identied, discontinuous impulsive (such as kicking a ball) and continuous (such pushing against a wall or lifting an entity). Using these scenarios, the authors suggest how network latency affects the perceived virtual entitys stiffness within the three application contexts identied. They also suggest appropriate distribution choices depending on the synchronisation demands of the application, overall favouring a client-server approach. In the case of static virtual environments, the authors argue in favour of a clientserver architecture if each user requires awareness of the others actions or peer-topeer otherwise. For applications employing turn taking, they suggest two alternatives client-server or token-ring. In the client-server case they suggest locking entities using a rst come rst served decision, while in the token-ring case they discuss ownership of entities according to pre-dened rules. The authors also suggest the use of an adaptive dynamic bounder which uses an estimate on the client side of when the next update will arrive, and alters the simulation dynamics according to a servers knowledge of the simulation and round-trip time. They also hypothesise about the possible use of a similar motion bounder in which the users allowable motion is adjusted to compensate for latency. In the case of applications requiring co-operative manipulation, they argue that a client-server architecture is essential to enforce consistency, suggesting the use of their adaptive dynamic bounder to cope with the round-trip latency.

14.3

Hybrid Peer-to-Peer

Hespanha et al. [HMS 00] suggest the use of collaborative haptics in a virtual museum application, in which participants can touch and share exhibits which cannot be normally handled in a real museum. Their proposed distribution architecture partitions the haptic environment into nodes, each consisting of a computer and operator. A dynamic node database is maintained which contains logical IDs, the IP addresses of all the connected nodes, as well as the latency and bandwidth available between them. Simulations are then replicated at each node, with entities locked to specic nodes. State is synchronised periodically through broadcasts. The authors state their solution is not very scalable, particularly when the simulation contains a large number of dynamic entities. They propose partitioning the nodes into local groups sharing a high bandwidth low latency connection to address this problem. Consequently participants on a local group are able to send more frequent updates between each other. Shen et al [SBNG03] suggest an alternative architecture for collaborative haptic audio visual environments (C-HAVE), based on the High Level Architecture (HLA) 54

Figure 43: A collaborative assembly task with haptic feedback and Run Time Interface (RTI) for parallel and distributed simulation [Dah99]. HLA uses the concept of federations which groups together interacting simulations. Synchronisation between these simulations is managed through the RTI. Rules relating to the ownership of entities in the simulation, dene how state is synchronised between connected peers. The C-HAVE system partitions the environment into federates which own regions of interest in which haptic interaction may take place. Each participant subscribes to a region when they enter it and discover the entities available. State information is then shared between subscribed participants. The authors suggest this architecture is potentially suitable for e-commerce applications.

14.4

Hybrid Client-Server

Marsh et al [MGP 04, MGPH05] dene and use the roaming-server architecture described earlier in 12.3.3 to manage haptic and non-haptic collaborative CAD prototyping over high latency high jitter networks. Figure 43 shows the Divipro application being used to perform an assembly sequence with haptic feedback through a Sensable PHANTOM device. Smoothing techniques, within the context of the Deva framework, to mitigate the effects of latency were illustrated using the application [MGP 04] and tested between Manchester and Labein. In subsequent work, the use of a haptic demon [MGPH05] to enable collaborative and co-operative hapic manipulation is described. The Divipro application provided the best low-latency haptic experience for collaborative interaction tasks, while still allowing co-operative manipulation. However, depending on network conditions, co-operative haptic manipulation was stable but not ideal for participants remote to the roamed server. Ongoing research into co-operative haptic manipulation, based on Hubbolds stretcher carrying task [Hub02] is currently being conducted at Manchester. A new implementation of the stretcher task, shown in Figure 44, borrows the concepts of the roaming server and haptic demon from Divipro in the design of distribution policies. The application also uses a more complex simulation of motion, and consequently has a high synchronisation demand. It is currently being tested over a range of different network connections. 55

Figure 44: A co-operative stretcher carrying task

14.5

Commercial Solutions

TiDeC a commercial solution, marketed by Handshake VR, to mitigate the effects of latency uses predictive methods to compensate for network delay. The system appears to compensate for delay by proling the network and predicting the typical delay between each connected participant [AZWS04], once this has been established predictive ltering methods are employed to compensate for the latency [AB95]. Being a commercial solution, little public technical information is available, however the company suggest their solution can compensate for latencies as great as 500ms. A number of pilot studies of the effects of latency on task performance have employed the system, to evaluate whether time-delay compensation can facilitate meaningful collaboration [WTR 03, AZWS04]. Completion times were measured for simple target acquisition tasks, and the authors report that increased latency in visual and haptic feedback adversely affects task completion.

15

Putting it All Together

It is clear that both performance and network limitations impose restrictions on the complexity of highly interactive shared virtual environments. Furthermore any single distribution architecture is unlikely to support the demands of every collaborative application possible. We have argued that hybrid architectures can be more exibly exploited to facilitate haptic interaction in behaviour-rich environments. However, there is still no clear general purpose solution to the perennial problem of high latency and high jitter connections. Consequently it is important to determine the type of interaction and the methods to mediate that, within the application context to best decide distribution strategy. In these notes we have highlighted potential problems that need consideration when choosing a distribution policy, and surveyed a number of solutions within the context of shared haptic interaction. Preferred strategies for interaction in shared virtual environments are application and task dependent, and for many applications collaborative (turn-taking) will sufce. While this does simplify state synchroni-

56

sation, additional cues to mediate transfer of control in the face of latency are essential to facilitate useful collaboration. In shared applications which require co-operative haptic manipulation, hybrid distribution choices must be coupled with techniques to maintain stable haptic interaction. One of the most promising techniques to achieve complex interaction in high quality, behaviour-rich shared virtual environments is to use selective methods exploiting human perception. These methods have been explored in isolation but rarely combined to implement perceptually driven behaviour-rich environments.

15.1

Perceptually Based Distribution

Recall that Devas [PCMW00] object/subject model for distributing rich behaviour decouples the real state of entities from the perceptual representation. The systems authors recognised that perception plays an important role in simplifying distribution strategies to facilitate shared interaction [Pet99, Mar02]. Central to the Deva approach is Immanuel Kants philosophy that while we believe our experience of the world to fully objective and un-mediated, it is straightforward to show that our perception of it can easily break down. For example optical illusions such as mirages present an immediate example of our perception failing to comprehend the world around us [Pet99, Mar02]. Kant describes a reality model in which a distinction is drawn between things themselves (he calls the noumenal reality), and things I perceive (which he calls the phenomenal reality). In the context of a virtual environment this model maps to the fact that entities in the environment possess state, and fundamental behaviour, that dene their semantic roles. Users in the environment experience this behaviour, by observing the changes in an entitys directly perceivable properties. Therefore, if an entity stores attribute internally as part of its simulated behaviour, but does not exhibit it in some manner it is of no relevance to the user and does not require distributing [Pet99,Mar02]. An early version of Deva simulated the objective reality on a parallel cluster known collectively as the Deva Server with rendering performed by dedicated clients called visualisers. These presented a view of the world and were treated as an extension of Kants human perceptual pipeline. Only changes in attributes that can ultimately be perceived (by affecting the 3D properties of an entity) were sent across the network. Perceptual lters were used to avoid sending unnecessary updates where the effect would be so small that it would be imperceptible. However, although the system was used to demonstrate the advantages of decoupling the objective and subjective realities for distribution, in practise strictly enforcing distribution of purely perceptible state information was found to be insufcient for maintaining perceptual coherency [Pet99, Mar02]. Consequently further versions of the system relaxed this restriction, while still employing the object/subject model for intelligent local behaviour simulation managed by the servers view of state. This still draws the distinction between the true state, and the perceived state of entities in the environment while allowing more exible state information exchange [Mar02]. Perceptually guided distribution choices offer the advantage of being able to reduce the state transfer demands of traditionally synchronisation intensive simulations. Approaches based on regions of interest [BBFG94, GB95, GFPB02], and visual attention 57

models to partition the distribution problem according to human visual perception have been successfully exploited [BWH03].

15.2

Graphical, Behavioural and Haptic Correspondence

Systems incorporating high quality graphics, complex behavioural simulations, and haptic feedback often need to maintain different data relating to each. In shared virtual environments this graphical, behavioural and haptic correspondence must additionally be shared between all connected participants. Potentially this can require a substantial amount of state information to be shared. From a rendering point of view, data describing geometry and material reectance characteristics must be maintained. Simulations also require models to represent their behavioural responses, and these may in fact be very different to the rendered geometry. For example an avatar may be rendered using surface geometry, while the motion of its body may be simulated using a system of interconnected links and joints. Further, the simulation model itself may be a simplied version of a realistic full model of the human bodys bones and joints. Similarly the required update rate for haptic rendering imposes a restriction on the complexity of models that can be displayed. Only a limited number of collision and response calculations can be performed within this tightly constrained update requirement. For this reason, it is necessary to simplify the geometry for haptic rendering. A number of methods have been employed to reduce the complexity of the problem space for haptic rendering, including partitioning environments and caching geometry in the region of interest [GHL05], and sensation preserving perceptual level-of-detail [OL03]. The latter technique exploits the fact that in many cases, graphical feedback plays a dominant role in user perception of the environment. Consequently as long as haptic feedback supports and re-enforces user expectations from the graphical display, the correspondence between underlying models need not match exactly. By maintaining intact key surface features while simplifying geometry using mesh decimation techniques, it is possible to selectively maintain enough perceptually important features for correct haptic perception.

15.3

Providing Richer Haptic Response

To provide richer haptic response in single user and shared virtual environments, it is not only important to maintain a perceptually acceptable mapping between models for graphical and haptic displays, but also use suitable force models to support correct perception of surface detail, entity-entity collisions, and motion during interactive manipulation tasks [OL03,OJSL04]. In the following sections the state-of-the-art methods supporting perceptually correct haptic interaction is detailed.

Acknowledgements
We would like to gratefully acknowledge the contributions of Dr. Steve Pettifer, Dr. James Marsh, Prof. Roger Hubbold and Ms. Caroline Jay without whom these course

58

notes would not have been possible. Dr. James Marsh in particular, has provided much of the content on distributed systems architectures, and the Deva System. We would also like to thank our partners, in particular Teresa Gutierrez at Labein; Holger Weiss at DLR; Gunter Kadegge at KL Technik; Ernesto Arenaza at Sener; Bas Ruiter at FCS Control Systems; and Matt Aranha at The University of Bristol for invaluable help with proling typical networks.

59

References
[AB95] A ZUMA R. T., B ISHOP G.: A frequency domain analysis of headmotion prediction. In SIGGRAPH Computer Graphics (1995), pp. 410 408. A DAMS R. J., K LOWDEN D., H ANNAFORD B.: Virtual training for a manual assembly task. Haptics-e 2, 2 (October 2001). A RSENAULT R., WARE C.: Eye-hand co-ordination with force feedback. In SIGCHI Conference on Human Factors in Computing Systems (The Hague, The Netherlands, 2000), pp. 408414. A LLISON R. S., Z ACHER J. E., WANG D., S HU J.: Effects of network delay on a collaborative motor task with telehaptic and televisual feedback. In ACM SIGGRAPH International Conference on Virtual Reality Continuum and its Applications in Industry (Singapore, 2004), ACM Press, pp. 375381. B ENFORD S., B OWERS J., FAHLN L. E., G REENHALGH C.: Managing mutual awareness in collaborative virtual environments. In Virtual Reality Software and Technology (Singapore, 1994), pp. 223236. B ROWN E., C AIRNS P.: A grounded investigation of game immersion. In Conference on Human Factors in Computing Systems (CHI) extended abstracts on Human factors in computing systems (Vienna, Austria, 2004), pp. 12971300. B OUVIER E., C OHEN E., NAJMAN L.: From crowd simulation to airbag deployment: Particle systems, a new paradigm of simulation. Electronic Imaging 6, 1 (1997), 94107. BASERMANN A., F INGBERG J., L ONSDALE G., M AERTEN B., WAL SHAW C.: Dynamic multi-partitioning for parallel nite element applications. Parallel Computing 27 (2001), 869881. B B. W., F RIEDMAN A., M C G AFFEY A.: Measuring and predicting visual delity. In SIGGRAPH (2001), ACM Press, pp. 213220. B ENFORD S. D., G REENHALGH C. M., L LOYD D.: Crowded collaborative virtual environments. In ACM Conference on Human Factors (CHI) (Atlanta, Georgia, 1997). B LAU B., H UGHES C. E., M OSHELL J. M., L ISLE C., O RLANDO F.: Networked virtual environments. In Siggraph Special Issue: Symposium on Interactive 3D Graphics (Cambridge MA, 1992), pp. 157164. BASDOGAN C., H O C. H., S RINIVASAN M. A.: Virtual environments for medical training: Graphical and haptic simulation of laparoscopic common bile duct exploration. IEEE/ASME Transactions on Mechatronics 6, 3 (September 2001), 269285. 60

[AKH01] [AW00]

[AZWS04]

[BBFG94]

[BC04]

[BCN97]

[BFL 01]

[BFM01] [BGL97]

[BHM 92]

[BHS01]

[BHSS00]

BASDOGAN C., H O C.-H., S HRINIVASAN M. A., S LATER M.: An experimental study on the role of touch in shared virtual environments. ACM Transactions on Computer Human Interaction 7, 4 (December 2000), 443460. B REEN D. E., H OUSE D. H., W OZNY M. J.: Predicting the drape of woven cloth using interacting particles. In Twenty First Annual Conference on Computer Graphics and Interactive Techniques (1994), ACM Press, pp. 365372. B LINN J.: Light reection functions for simulation of clouds and dusty surfaces. Computer Graphics 16, 3 (July 1982), 2129. B OLIN M. R., M EYER G. W.: A perceptually based adaptive sampling algorithm. In SIGGRAPH Computer Graphics (July 1998), pp. 299 310. B RO -N IELSEN M.: Finite element modeling in surgery simulation. In IEEE (March 1998), vol. 86, pp. 490503. B UTTOLO P., O BOE R., H ANNAFORD B.: Architectures for shared haptic virtual environments. Special Issue of Computers and Graphics (1997).

[BHW94]

[Bli82] [BM98]

[BN98] [BOH97]

[BOYBK90] B ROOKS F. P., O UH -YOUNG M., BATTER J. J., K ILPATRICK P. J.: Project GROPE haptic displays for scientic visualization. In ACM SIGGRAPH Computer Graphics (August 1990), vol. 24 (4), pp. 177 185. [Bro99] [BRS 03] B ROOKS J R F. P.: Whats real about virtual reality. IEEE Computer graphics and applications (1999), 1627. BARANOSKI G. V. G., ROKNE J. G., S HIRLEY P., T RONDSEN T. S., BASTOS R.: Simulating the aurora. The Journal of Visualization and Computer Animation 14, 1 (February 2003), 4359. B ERETSEKAS D., T SITSIKLIS J.: Parallel and Distributed Computation Numerical Methods. Prentice Hall, 1989. B URDEA G. C.: Force and Touch Feedback for Virtual Reality. John Wiley and Sons, Inc., 1996. B UTCHER J. C.: The Numerical Analysis of Ordinary Differential Equations: Runge-Kutta and General Linear Methods. WileyInterscience, New York, 1987. B LOOMENTHAL J., W YVILL B. (Eds.): Introduction to Implicit Surfaces. Barnes & Noble, 1997. BARAFF D., W ITKIN A.: Large steps in cloth simulation. In SIGGRAPH (1998), pp. 4354. 61

[BT89] [Bur96] [But87]

[BW97] [BW98]

[BWH03]

B EEHAREE A. K., W EST A. J., H UBBOLD R.: Visual attention based information culling for distributed virtual environments. In Tenth ACM Symposium on Virtual Reality Software and Technology (VRST) (October 2003), ACM Press, pp. 213222. BARAFF D., W ITKIN A., K ASS M.: Untangling cloth. In SIGGRAPH (July 2003), vol. 22, ACM Press, pp. 862870. C ATER K., C HALMERS A. G., L EDDA P.: Selective quality rendering by exploiting human inattentional blindness: Looking but not seeing. In Symposium on Virtual Reality Software and Technology (2002), ACM, pp. 1724. C ATER K., C HALMERS A., WARD G.: Detail to attention: Exploiting visual tasks for selective rendering. In Eurographics Symposium on Rendering (Leuven, Belgium, 2003), Eurographics Association, pp. 270 280. C OTIN S., D ELINGETTE H., AYACHE N.: Real-time elastic deformations of soft tissues for surgery simulation. IEEE Transactions on Visualization and Computer Graphics 5, 1 (1999), 6273. C OULOURIS G., D OLLIMORE J., K INDBERG T.: Distributed Systems Concepts and Design, third ed. Addison Wesley, 2001. C HALMERS A., DAVIS T., R EINHARD E.: Practical Parallel Rendering. AK Peters, Nantick, USA, 2002. C ANI -G ASCUEL M.-P., D ESBRUN M.: Animation of deformable models using implicit surfaces. IEEE Transactions on Visualisation and Computer Graphics 3, 1 (March 1997), 3950. C HALMERS A. G., K K. C., M AFLIOLI D.: Visual attention models for producing high delity graphics efciently. In Spring Conference on Computer Graphics (Budermice, April 2003). C OOK J., P ETTIFER S.: Placeworld: An integration of shared virtual worlds. In ACM SIGGRAPH Sketches and applications (August 2001), ACM Press, p. 243. C ONWAY M., PAUSCH R., G OSSWEILER R., B URNETTE T.: Alice: A rapid prototyping system for building virtual environments. In ACM CHI Human Factors in Computing Systems (Boston, Massachusetts, 1994), pp. 295296. C OHEN M. F., WALLACE J. R., H ANRAHAN P.: Radiosity and Realistic Image Synthesis. Academic Press Professional, San Diego, CA, USA, 1993.

[BWK03] [CCL02]

[CCW03]

[CDA99]

[CDK01] [CDR02] [CGD97]

[CKM03]

[CP01]

[CPGB94]

[CWH93]

62

[Dah99]

DAHMANN J. S.: The high level architecture and beyond: Technology challenges. In Thirteenth Workshop on Parallel and Distributed Simulation (Atlanta, Georgia, United States, 1999), IEEE Computer Society, pp. 6470. DALY S.: The visible differences predictor: An algorithm for the assessment of image delity. In Digital Image and Human Vision (Cambridge, MA, 1993), Watson A. B., (Ed.), MIT Press, pp. 179206. D ELINGETTE H., C OTIN S., AYACHE N.: A hybrid elastic model allowing real-time cutting, deformations and force-feedback for surgery training and simulation. In Computer Animation (May 1999), pp. 70 81. D ESBRUN M., G ASCUEL M.-P.: Animating soft substances with implicit surfaces. In Twenty Second Annual Conference on Computer Graphics and Interactive Techniques (1995), ACM Press, pp. 287290. D OUGLAS Y., H ARGADON A.: The pleasure principle: Immersion engagement and ow. In Proceedings of Hypertext (2000), ACM Press, pp. 153160. D INH H. Q., WALKER N., S ONG C., KOBAYASHI A., H ODGES L. F.: Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In IEEE Virtual Reality (1999), pp. 222228. E MLING J. W., M ITCHELL D.: The effects of time delay and echos on telephone conversations. Bell System Technical Journal 22 (November 1963), 28692891. E NRIGHT D., M ARSCHNER S., F EDKIW R.: Animation and rendering of complex water surfaces. ACM Transactions on Graphics 21, 3 (July 2002), 736744.

[Dal93]

[DCA99]

[DG95]

[DH00]

[DWS 99]

[EM63]

[EMF02]

[EMTTT98] E ARNSHAW R., M AGNENAT-T HALMANN N., T ERZOPOULOS D., T HALMANN D.: Guest editors introduction: Computer animation for virtual humans. IEEE Computer Graphics and Applications 18, 5 (SeptemberOctober 1998), 2023. [EPMW99] E CONOMOU D., P ETTIFER S. R., M ITCHELL W. L., W EST A. J.: The Kahun project: CVE technology development based on real world application and user needs. In ACM Symposium on Virtual Reality Software and Technology (1999), pp. 168169. F ERRELL W. R.: Delayed force feedback. Human Factors 8 (1966), 449455.

[Fer66]

63

[Fit54]

F ITTS P. M.: The information capacity of the human motor system in controlling the amplitude of movement. Experimental Psychology 47, 6 (1954), 381391. F ERWERDA J., PATTANAIK S. N., S HIRLEY P., G REEENBERG D. P.: A model of visual adaptation for realistic image synthesis. In SIGGRAPH (1996), ACM Press, pp. 249258. F OURNIER A., R EEVES W. T.: A simple model of ocean waves. In ACM SIGGRAPH Computer Graphics (August 1986), ACM Press, pp. 7584. F RCON E., S TENIUS M.: DIVE: A scaleable network architecture for distributed virtual environments. Distributed Systems Engineering Special Issue on Distributed Virtual Environments 5, 3 (September 1998), 91100. F EDKIW R., S TAM J., J ENSEN H. W.: Visual simulation of smoke. In SIGGRAPH Computer Graphics (2001), ACM Press, pp. 1522. F UJIMOTO R. M.: Parallel and Distributed Simulation Systems. WileyInterscience, 1999. G REENHALGH C., B ENFORD S.: MASSIVE: A collaborative virtual environment for teleconferencing. ACM Transactions on ComputerHuman Interaction (TOCHI) 2, 3 (September 1995), 239261. G IBSON S., C OOK J., H OWARD T. L. J., H UBBOLD R. J.: Rapid shadow generation in real-world lighting environments. In Eurographics Symposium on Rendering (Leuven, Belgium, June 2003). G REENHALGH C., F LINTHAM M., P URBRICK J., B ENFORD S.: Applications of temporal links: Recording and replaying virtual environments. In IEEE Virtual Reality Conference (Orlando, Florida, March 2002), pp. 101108. G IBSON S., H UBBOLD R. J.: Perceptually-driven radiosity. Computer Graphics Forum 16, 2 (June 1997), 129141. G UNN C., H UTCHINS M., A DCOCK M., H AWKINS R.: Trans-world haptic collaboration. In SIGGRAPH: Sketches and Applications (2003). G LENCROSS M., H UBBOLD R., LYONS B.: Dynamic primitive caching for haptic rendering of large-scale models. In IEEE worldHAPTICS First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (Pisa, Italy, March 2005), pp. 517518. G LENCROSS M., H OWARD T., P ETTIFER S.: Iota: An approach to physically-based modelling in virtual environments. In IEEE Virtual Reality Conference (Yokohama, Japan, March 2001), pp. 287288. 64

[FPSG96]

[FR86]

[FS98]

[FSJ01] [Fuj99] [GB95]

[GCHH03]

[GFPB02]

[GH97] [GHAH03] [GHL05]

[GHP01]

[GHSA05]

G UNN C., H UTCHINS M., S TEVENSON D., A DCOCK M.: Using collaborative haptics in remote surgical training. In worldHAPTICS First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (Pisa, Italy, March 2005), IEEE Computer Society, pp. 481482. G LASSNER A. S.: An Introduction to Ray Tracing. Academic Press, London, UK, 1989. G LENCROSS M.: A Framework for Physically Based Modelling in Virtual Environments. PhD thesis, The University of Manchester, 2000. G LENCROSS M., M URTA A.: Multi-body simulation in virtual environments. In Simulation Past, Present and Future. Twelveth European Simulation Multiconference (Manchester, England, June 1998), Zobel R., Moeller D., (Eds.), pp. 590594. G LENCROSS M., M URTA A.: A virtual jacobs ladder. In Graphicon (Moscow, August 1999), pp. 8894. G LENCROSS M., M URTA A.: Managing the complexity of physically based modelling in virtual reality. In Fourth International Conference of Computer Graphics and Articial Intelligence (Limoges, May 2000), pp. 4148. G UPTA R., S HERIDAN T., W HITNEY D.: Experiments using multimodal virtual environments in design for assembly analysis. Presense: Teleoperators and Virtual Environments 6, 3 (1997), 318338. G REENBERG D. P., T ORRANCE K. E., S HIRLEY P., A RVO J., F ER WERDA J., PATTANAIK S. N., L AFORTUNE A. E., WALTER B., F OO S.-C., T RUMBORE B.: A framework for realistic image synthesis. In SIGGRAPH: Special Session (1997), ACM Press, pp. 477494. G UTWIN C.: The effects of network delays on group work in real-time groupware. In European Conference on CSCW (Bonn, 2001), pp. 299 318. H AGSAND O.: Interactive multiuser VEs in the DIVE system. IEEE Multimedia 3, 1 (1996), 3039. H AND C.: A survey of 3d interaction techniques. Computer Graphics Forum 16, 5 (December 1997), 269281. H ARRIS M.: Fast uid dynamics simulation on the GPU. In GPU Gems: Programming Techniques, Tips, and Tricks for Real-Time Graphics, Fernando R., (Ed.). Addison-Wesley, 2004, pp. 637665.

[Gla89] [Gle00] [GM98]

[GM99] [GM00]

[GSW97]

[GTS 97]

[Gut01]

[Hag96] [Han97] [Har04]

65

[HBSL03]

H ARRIS M. J., BAXTER W. V., S CHEUERMANN T., L ASTRA A.: Simulation of cloud dynamics on graphics hardware. In SIGGRAPH/Eurographics Workshop On Graphics Hardware (San Diego, California, 2003), ACM Press, pp. 91101. H ARRIS M. J., C OOMBE G., S CHEUERMANN T., L ASTRA A.: Physically-based visual simulation on graphics hardware. In SIGGRAPH/Eurographics Workshop On Graphics Hardware (2002), ACM Press. H ECKBERT P. S.: Simulating Global Illumination Using Adaptive Meshing. PhD thesis, University of California at Berkeley, 1992. H ASSER C. J., G LODENBURG A. S., M ARTIN K. M., ROSENBURG L. B.: User performing a GUI pointing task with a low-cost forcefeedback computer mouse. In ASME Dynamic Systems and Control Division (1998), vol. 64, pp. 151156. H ARRIS M. J., L ASTRA A.: Real-time cloud rendering. In Eurographics (2001), Blackwells, pp. 7684. H UANG G., M ETAXAS D., G OVINDARAJ M.: Feel the "fabric": An audio-haptic interface. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation (2003), pp. 5261. H ESPANHA J. P., M C L AUGHLIN M., S UKHATME G. S., A KBARIAN M., G ARG R., Z HU W.: Haptic collaboration over the internet. In The Fifth PHANTOM Users Group Workshop (2000). H ALLORAN J., ROGERS Y., F ITZPATRICK G.: From text to talk: Multiplayer games and voiceover IP. In Level Up: First International Digital Games Research Conference (2003), pp. 130142. H UBBOLD R. J.: Collaborative stretcher carrying: A case study. In ACM Workshop on Virtual Environments (Barcelona, Spain, 2002), pp. 712. I TTI L., KOCH C.: A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research 40, 1012 (2000), 14891506. I TTI L., KOCH C., N IEBUR E.: A model of saliency-based visual attention for rapid scene analysis. Pattern Analysis and Machine Intelligence 20 (1998), 12541259. JAMES W.: Principles of Pshychology. Holt, New York, 1890. J EFFERSON D. R.: Virtual time. ACM Transactions on Programming Languages and Systems 7, 3 (1985), 404425.

[HCSL02]

[Hec92] [HGMR98]

[HL01] [HMG03]

[HMS 00]

[HRF03]

[Hub02]

[IK00]

[IKN98]

[Jam90] [Jef85]

66

[Jen01] [JH04]

J ENSEN H. W.: Realistic Image Synthesis Using Photon Mapping. AK Peters, Nantick, USA, 2001. JAY C., H UBBOLD R.: Does performance in the face of delayed sensory feedback change with practice? In Eurohaptics (2004), Springer-Verlag, pp. 530533. ISBN 3-9809614-1-9. JAY C., H UBBOLD R.: Delayed visual and haptic feedback in a reciprocal tapping task. In IEEE worldHAPTICS First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (Pisa, Italy, March 2005), pp. 655656. ISBN 0-7695-2310-2/05. Jorvik: The Viking city. Online document, May 2005. http://www.jorvik-viking-centre.co.uk/jorvik-navigation.htm. K HOTE M., H OWARD T.: Mirages. In Eurographics UK Fifteenth Annual Conference (Norwich, England, March 1997), pp. 223235. K IM J., K IM H., TAY B. K., M UNIYANDI M., S RINIVASAN M. A., J ORDAN J., M ORTENSEN J., O LIVEIRA M., S LATER M.: Transatlantic touch: A study of haptic collaboration over long distance. Presence: Teleoperator and Virtual Environments 13, 3 (2004), 328337. K USCHFELDT S., S CHULZ M., E RTL T., R EUDING T., H OLZNER M.: The use of a virtual environment for FE analysis of vehicle crashworthiness. In Virtual Reality Annual International Symposium VRAIS (Albuquerque, USA, March 1997), p. 209. K ALLMANN M., T HALMANN D.: Direct 3D interaction with smart objects. In VRST (London, December 1999), ACM Press. KOMERSKA R., WARE C.: Haptic task constraints for 3D interaction. In IEEE Symposium on Virtual Reality (2003). K UNIMATSUA A., WATANABE Y., F UJII H., S AITO T., H IWADA K., TAKAHASHI T., U EKI H.: Rendering of natural phenomena fast simulation and rendering techniques for uid objects. Computer Graphics Forum 20, 3 (2001), 5767. L ABEIN: DATum geometric modeller PDriver users manual. Version 15.0, 1999. L ONGHURST P., D EBATTISTA K., C HALMERS A.: Snapshot: A rapid technique for selective global illumination rendering. In Thirteenth International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (2005). L UBEKE D., H ALLEN B.: Perceptually driven simplication for interactive rendering. In Twelveth Eurographics Workshop on Rendering (2001), pp. 221223. 67

[JH05]

[Jor05] [KH97] [KKT 04]

[KSE 97]

[KT99] [KW03] [KWF 01]

[Lab99] [LDC05]

[LH01]

[LLW04]

L IU Y., L IU X., W U E.: Real-time 3D uid simulation on GPU with complex obstacles. In Computer Graphics and Applications Twelveth Pacic Conference (2004), pp. 247256. L OSCHKY L. C., M C C ONKIE G. W.: Gaze contingent displays: Maximizing display bandwidth efciency. In ARL Federated Laboratory Advanced Displays and Interactive Displays Consortium, Advanced Displays and Interactive Displays Third Annual Symposium (1999), pp. 79 83. L OSCHKY L. C., M C C ONKIE G. W., YANG J., M ILLER M. E.: Perceptual effects of a gaze-contingent multi-resolution display based on a model of visual sensitivity. In ARL Federated Laboratory 5th Annual Symposium - ADID Consortium (2001), pp. 5358. L UBEKE D., R EDDY M., WATSON B., C OHEN J., VARSHNEY A.: Advanced issues in level of detail. In SIGGRAPH: Course (Los Angeles, August 2001), ACM Press. L ARSSON P., VSTFJLL D., K LEINER M.: Better presence and performance in virtual environments by improved binaural sound rendering. In AES Twenty Second International Conference on Virtual Synthetic and Entertainment Audio (Espoo, Finland, June 2002), pp. 3138. M ANNINEN T.: Rich interaction in the context of networked virtual environments - experiences gained from the multi-player games domain. In Joint HCI and IHM Conference (2001), Blanford A., Vanderdonckt J. Gray P., (Eds.), pp. 383398. M ANNINEN T.: Virtual team interactions in networked multimedia games case: Counter-Strike Multi-player 3D action game. In Presence Conference (Phiadelphia, May 2001). M ARSH J.: A Software Architecture for Interactive Multiuser Visualisation. PhD thesis, The University of Manchester, 2002. M AUVE M.: Consistency in replicated continuous interactive media. In ACM Conference on Computer Supported Cooperative Work (CSCW) (Philadelphia, PA, December 2000), pp. 181190. M ONTGOMERY K., B RUYNS C., B ROWN J., S ORKIN S., M AZZELLA F., T HONIER G., T ELLIER A., L ERMAN B., M ENON A.: Spring: A general framework for collaborative, real-time surgical simulation. In Medicine Meets Virtual Reality (Amsterdam, 2002), IOS Press. M USSE S. R., BABSKI C., C APIN T., T HALMANN D.: Crowd modelling in collaborative virtual environments. In VRST (Taipei, Taiwan, November 1998), pp. 115124.

[LM99]

[LMYM01]

[LRW 01]

[LVK02]

[Man01a]

[Man01b]

[Mar02] [Mau00]

[MBB 02]

[MBCT98]

68

[MCTG00]

M C N AMARA A., C HALMERS A., T ROSCIANKO T., G ILCHRIST I.: Comparing real and synthetic scenes using human judgements of lightness. In Eurographics Workshop on Rendering (Brno, June 2000). M ARMITT G., D UCHOWSKI A. T.: Modeling visual attention in VR: Measuring the accuracy of predicted scanpaths. In Eurographics: Short Presentations (2002), pp. 217226. M OTTET D., G UIARD Y., F ERRAND T., B OOTSMA R. J.: Two-handed performance of a rhythmical tts task by individuals and dyads. Experimental Psychology: Human Perception and Performance 27, 6 (2001), 12751286. M ARSH J., G LENCROSS M., P ETTIFER S., H UBBOLD R., C OOK J., DAUBRENET S.: Minimising latency and maintaining consistency in distributed virtual prototyping. In ACM SIGGRAPH Conference on the Virtual Reality Continuum and its Applications in Industry (VRCAI) (Singapore, June 2004), pp. 386389. M ARSH J., G LENCROSS M., P ETTIFFER S., H UBBOLD R.: A robust network architecture supporting rich behaviour in collaborative interactive applications. In Review for IEEE TVCG Special Issue on Special Issue-Haptics, Virtual and Augmented Reality (2005). M ICHOTTE A. E.: The Perception of Causality. Basic Books, 1963. M IRTICH B.: Timewarp rigid body simulation. In ACM SIGGRAPH Computer Graphics (New Orleans, LA, 2000), pp. 193200. M EEHAN M., I NSKO B., W HITTON M., J R . F. P. B.: Physiological measures of presence in stressful virtual environments. In ACM SIGGRAPH Transactions on Graphics (2002), pp. 645652. M C C ONKIE G. W., L OSCHKY L. C.: Human performance with a gazelinked multi-resolutional display. In ARL Federated Laboratory Advanced Displays and Interactive Displays Consortium, Advanced Displays and Interactive Displays First Annual Symposium (1997), pp. 25 34. M ARCELINO L., M URRAY N., F ERNANDO T.: A constraint manager to support virtual maintainability. In Computers and Graphics (2003), vol. 27, pp. 1926. M USGRAVE F. K., P EACHEY D., P ERLIN K., W ORLEY S.: Texturing and Modeling A Procedural Approach. Academic Press Limited, 1994. M ARSH J., P ETTIFER S., W EST A.: A technique for maintaining continuity of experience in networked virtual environments. In UKVRSIG (Sept. 1999).

[MD02]

[MGFB01]

[MGP 04]

[MGPH05]

[Mic63] [Mir00] [MIWJ02]

[ML97]

[MMF03]

[MPPW94] [MPW99]

69

[MR98] [MR04]

M ACK A., ROCK I.: Inattentional Blindness. MIT Press, 1998. M ANIA K., ROBINSON A.: The effect of quality of rendering on user lighting impressions and presence in virtual environments. In ACM SIGGRAPH International conference on Virtual Reality Continuum and its Applications in Industry (VRCAI) (Singapore, 2004). M EYER G. W., RUSHMEIER H. E., C OHEN M. F., G REENBERG D. P., T ORRANCE K. E.: An experimental evaluation of computer graphics imagery. ACM Transactions on Graphics 5, 1 (1986), 3050. M EEHAN M., R AZZAQUE S., W HITTON , J R . F. P. B.: Effect of latency on presence in stressful virtual environments. In IEEE Virtual Reality (2003), pp. 141148. M ACIEL P. W. C., S HIRLEY P.: Visual navigation of large environments using textured clusters. In Symposium on Interactive 3D Graphics (1995), pp. 95102. M YSZKOWSKI K., TAWARA T., A KAMINE H., S EIDEL H.-P.: Perception-guided global illumination solution for animation rendering. In SIGGRAPH (2001), ACM Press, pp. 221230. M ANIA K., T ROSCIANKO T., H AWKES R., C HALMERS A.: Fidelity metrics for virtual environment simulations based on spatial memory awareness states. Presence: Teleoperators and Virtual Environments 12, 3 (2003), 296310. M USGRAVE F.: Prisms and rainbows: A dispersion model for computer graphics. In Proceedings of the Graphics Interface Vision Interface (Toronto, Ontario, June 1989). M AUVE M., VOGEL J., H ILT V., E FFELSBERG W.: Local-lag and timewarp: Providing consistency for replicated continuous applications. IEEE Transactions on Multimedia 6, 1 (February 2004), 4757. M AC K ENZIE I. S., WARE C.: Lag as a determinant of human performance in interactive systems. In INTERCHI Proceedings of the ACM Conference on Human Factors in Computing Systems (New York, 1993), pp. 488493. M IYAZAKI R., YOSHIDA S., D OBASHI Y., N ISHITA T.: A method for modelling clouds cased on atmospheric uid dynamics. In Pacic Graphics Conference (2001), IEEE Computer Society, pp. 363372. M YSZKOWSKI K.: The visible differences predictor: Applications to global illumination problems. In Eurographics Workshop on Rendering (1998), pp. 223226.

[MRC 86]

[MRWJ03]

[MS95]

[MTAS01]

[MTHC03]

[Mus89]

[MVHE04]

[MW93]

[MYDN01]

[Mys98]

70

[MZP 94]

M ACEDONIA M. R., Z YDA M. J., P RATT D. R., BARHAM P. T., Z ESWITZ S.: NPSNET: A network software architecture for large-scale ves. Presence Teleoperators and Virtual Environments 3, 4 (1994), 265 287. OTADUY M. A., JAIN N., S UD A., L IN M. C.: Haptic display of interaction between textured models. Proc. of IEEE Visualization (2004), 297304. OTADUY M. A., L IN M. C.: Sensation preserving simplication for haptic rendering. In ACM SIGGRAPH Transactions on Graphics (San Diego, CA, 2003), vol. 22, pp. 543553. O LIVEIRA M., M ORTENSEN J., J ORDAN J., S TEED A., S LATER M.: Considerations in the design of virtual environment systems: A case study. In Second International Conference on Application and Development of Computer Games (Hong Kong, January 2003). P OPE J., C HALMERS A.: Multi-sensory rendering: Combining graphics and acoustics. In Seventh International Conference in Central Europe on Computer Graphics (1999). P ETTIFER S., C OOK J., M ARIANI J., T REVOR J.: Exploring realtime visualisation of large abstract data spaces with QSPACE. In IEEE Virtual Reality (Yokohama, Japan, March 2001), pp. 293294. P ETTIFER S., C OOK J., M ARSH J., W EST A.: DEVA3: Architecture for a large-scale distributed virtual reality system. In ACM Symposium on Virtual Reality Software and Technology (Seoul, Korea, 2000), pp. 3340. P ETTIFER S. R.: An operating environment for large scale virtual reality. PhD thesis, The University of Manchester, 1999. PATTANAIK S. N., FAIRCHILD J. F. M. D., G REENBERG D. P.: A multiscale model of adaptation and spatial vision for realistic image display. In SIGGRAPH (1998), ACM Press, pp. 287298. PARK K. S., K ENYON R. V.: Effects of network characteristics on human performance in a collaborative virtual environment. In IEEE Virtual Reality (1999), IEEE Computer Society, pp. 104111. P ETTIFER S., M ARSH J.: A collaborative access model for shared virtual environments. In Tenth IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises WETICE (2001), pp. 257262. PARKER S., M ARTIN W., S LOAN P. P., S HIRLEY P., S MITS B., H ANSEN C.: Interactive ray tracing. In Symposium on Interactive 3D Computer Graphics (April 1999). 71

[OJSL04]

[OL03]

[OMJ 03]

[PC99]

[PCMT01]

[PCMW00]

[Pet99] [PFG98]

[PK99]

[PM01]

[PMS 99]

[RAB 00]

ROSE F. D., ATTREE E. A., B ROOKS B. M., PARSLOW D. M., P ENN P. R.: Training in virtual environments: Transfer to real world tasks and equivalence to real task training. Ergonomics 43, 4 (April 2000), 494511. R EDDY M.: Perceptually Modulated Level of Detail for Virtual Environments (CST- 134-97). PhD thesis, University of Edinburgh, 1997. R EINER M.: The role of haptics in immersive telecommunication environments. IEEE Transactions on Circuits and Systems for Video Technology 14, 3 (March 2004), 392401. R EYNOLDS C. W.: Flocks, herds, and schools: A distributed behavioral model. In SIGGRAPH Computer Graphics (1987), vol. 21, pp. 2534. ROESSLER A., G RANTZ V.: Performance evaluation of input devices in virtual environments. In IFIP Working Group Conference on Designing Effective and Usable Multimedia Systems (1998), pp. 6878. ROLLAND J. P., G IBSON W., A RIELY D.: Towards quantifying depth and size perception in virtual environments. Presence: Teleoperators and Virtual Environments 4, 1 (1995), 2449. RUHLEDER K., J ORDAN B.: Meaning-making across remote sites: How delays in transmission affect interaction. In Sixth European Conference on Computer-Supported Cooperative Work ECSCW (Copenhagen, Denmark, 1999), Kulwer, pp. 411429. R IESZ R., K LEMMER E. T.: Subjective evaluation of delay and echo supressors in telephone communications. Bell System Technical Journal 42 (November 1963), 29192933. RUSHMEIER H., L ARSON G., P IATKO C., S ANDERS P., RUST B.: Comparing real and synthetic images: Some ideas about metrics. In Eurographics Rendering Workshop (June 1995), pp. 8291. R AMASUBRAMANIAN M., PATTANAIK S. N., G REENBERG D. P.: A perceptually based physical error metric for realistic image synthesis. In SIGGRAPH (1999), ACM Press, pp. 7382. S HEN X., B OGSANYI F., N I L., G EORGANAS N. D.: A heterogeneous scalabale architecture for collaborative haptics environments. In IEEE International Workshop on Haptic Audio and Visual Environments (HAVE) (September 2003), pp. 113118. S IMONS D., C HABRIS C.: Gorillas in our midst: Sustained inattentional blindness for dynamic events. Perception 28 (1999), 10591074. S UNDSTEDT V., C HALMERS A., C ATER K., D EBATTISTA K.: Topdown visual attention for efcient rendering of task related scenes. In Vision, Modeling and Visualization (2004). 72

[Red97] [Rei04]

[Rey87] [RG98]

[RGA95]

[RJ99]

[RK63]

[RLP 95]

[RPG99]

[SBNG03]

[SC99] [SCCD04]

[SDL 05]

S UNDSTEDT V., D EBATTISTA K., L ONGHURST P., C HALMERS A., T ROSCIANKO T.: Visual attention for efcient high-delity graphics. In Spring Conference on Computer Graphics SCCG (May 2005). S TAM J., F IUME E.: Turbulent wind elds for gaseous phenomena. In Computer Geraphics (August 1993), vol. 9, pp. 369376. S HIRMOHAMMADI S., G EORGANAS N. D.: An end-to-end communication architecture for collaborative virtual environments. Computer Networks 35, 2-3 (2001), 351367. S HAW J.: Distributed legible city. In IST 98 Information Society Technologies Conference and Exhibition (Vienna, Austria, 1998). S LATER M.: Measuring presence: A response to the witmer and singer presence questionnaire. Presence Teleoperators and Virtual Environments 8, 5 (1999), 560565. S ILLION F. X., P UECH C.: Radiosity and Global Illumination. Morgan Kaufmann, San Francisco, CA, USA, 1994. S ALLNS E.-L., R ASSMUS -G ROEHN K., S JOESTRM C.: Supporting presence in collaborative environments by haptic force feedback. ACM Transactions on Computer-Human Interaction 7, 4 (2000), 461476. S TEVENSON D. R., S MITH K. A., M C L AUGHLIN J. P., G UNN C. J., V ELDKAMP J. P., D IXON M. J.: Haptic workbench: A multisensory virtual environment. In SPIE (1999), vol. 3639, pp. 356366. S LATER M., S ADAGIC A., U SOH M.: Small group behaviour in a virtual and real time environment: A comparative study. Presence: Teleoperators and Virtual Environments 9, 1 (2000), 3751. S TAM J.: Interacting with smoke and re in real-time. Communications of the ACM 43, 7 (July 2000), 7683. S LATER M., W ILBUR S.: A framework for immersive virtual environments (FIVE): speculations on the role of presence in virtual environments. Presence: Teleoperators and Virtual Environments 6, 6 (1997), 603616. S INGHAL S., Z YDA M.: Networked Virtual Environments Design and Implementation. ACM Press SIGGRAPH Series Addison Wesley, 1999. S ALLS E.-L., Z HAI S.: Collaboration meets tts law: Passing virtual objects with and without haptic force feedback. In INTERACT IFIP Conference on Human-Computer Interaction (2003), pp. 97104. T ERZOPOULOS D., F LEISCHER K.: Modelling in-elastic deformation: Viscoelasticity, plasticity, fracture. In SIGGRAPH (1988), ACM Press, pp. 269278. 73

[SF93] [SG01]

[Sha98] [Sla99]

[SP94] [SRGS00]

[SSM 99]

[SSU00]

[Sta00] [SW97]

[SZ99] [SZ03]

[TF88]

[TF98] [TGD04]

T ERZOPOULOS D., F LEISCHER K.: Deformable models. Th Visual Computer 4 (1998), 306331. T SINGOS N., G ALLO E., D RETTAKIS G.: Perceptual audio rendering of complex virtual environments. In ACM SIGGRAPH Transactions on Graphics (August 2004), vol. 23 (3), pp. 249258. T ORKINGTON J., S MITH S., R EES B., DARZI A.: Skill transfer from virtual reality to a real laparoscopic task. Surgical Endoscopy 15, 10 (May 2001), 10761079. U LICNY B., T HALMANN D.: Crowd simulation for interactive virtual environments and VR training systems. In Eurographics workshop on Animation and Simulation (2001), Springer-Verlag, pp. 163170. U LICNY B., T HALMANN D.: Towards interactive real-time crowd behavior simulation. Computer Graphics Forum 21, 4 (December 2002), 767775. VOLEVICH V., M YSZKOWSKI K., K HODULEV A., KOPYLOV E. A.: Using the visual differences predictor to improve performance of progressive global illumination computations. ACM Transactions on Graphics 19, 2 (April 2000), 122161. Welding simulator. Online Document, http://www.vrsim.net/Products/weldingsimmain.html. May 2005.

[TSRD01]

[UT01]

[UT02]

[VMKK00]

[VRS05] [Wal05] [WDGT01]

WALLACE M.: Vrsim welding application. Personnal correspondance with CEO of VRSim, 2005. W U X., D OWNES M. S., G OTEKIN T., T ENDICK F.: Adaptive nonlinear nite elements for deformable body simulation using dynamic progressive meshes. In Eurographics (2001), Chalmers A., Rhyne T. M., (Eds.), vol. 20, pp. 349358. W EISS H.: Dynamics of geometrically nonlinear rods: I. mechanical models and equations of motion. Nonlinear Dynamics 30 (2002), 357 381. W EISS H.: Dynamics of geometrically nonlinear rods: II. mechanical models and equations of motion. Nonlinear Dynamics 30 (2002), 383 415. W INTER A. B. P. R.: Theory and application of nite element analysis to structural crash simulation. Computers and Structures 13 (June 1981), 277285. WALD I., KOLLIG T., B ENTHIN C., K ELLER A., S LUSALLEK P.: Interactive global illumination using fast ray tracing. In Thirteenth Eurographics workshop on Rendering (Pisa, Italy, 2002), Eurographics Association, pp. 1524. ray tracing, realistic rendering, real-time, global illumination. 74

[Wei02a]

[Wei02b]

[Win81]

[WKB 02]

[WKSKH99] W ILSON J. P., K LINE -S CHODER R. J., K ENTON M. A., H OGAN N.: Algorithms for network-based force feedback. In The Fourth PHANTOM Users Group Workshop (1999). [WS98] W ITMER B. G., S INGER M. J.: Measuring presence in virtual environments: A presence questionnaire. Presence Teleoperators and Virtual Environments 7, 3 (1998), 225240. WALD I., S LUSALLEK P., B ENTHIN C., WAGNER M.: Interactive rendering with coherent ray tracing. Computer Graphics Forum 20, 3 (2001). WANG D., T UER K., ROSSI M., N I L., S HU J.: The effect of time delays on tele-haptics. In Second IEEE Internatioal Workshop on Haptic, Audio and Visual Environments and Their Applications HAVE (2003), pp. 712. WATSEN K., Z YDA M.: Bamboo A portable system for dynamically extensible, real-time, networked, virtual environments. In Virtual Reality Annual International Symposium (1998), pp. 252259. YANTIS S.: Attentional capture in vision. In Converging Operations in the Study of Selective Visual Attention, Kramer A., Coles M. Logan G., (Eds.). American Psychological Association, 1996, pp. 4576. YARBUS A. L.: Eye movements during perception of complex objects. In Eye Movements and Vision (New York, 1967), Riggs L. A., (Ed.), Plenum Press, pp. 171196. Y EE H.: Spatiotemporal Sensitivity and Visual Attention for Efcient Rendering of Dynamic Environments. Masters thesis, Cornell University, 2000. Y EE H., PATTANAIK S., G REENBERG D. P.: Spatiotemporal sensitivity and visual attention for efcient rendering of dynamic environments. ACM Transactions on Computer Graphics 20, 1 (2001), 3965. Z HANG Y., F ERNANDO T.: 3D sound feedback act as task aid in a virtual assembly environment. In Theory and Practice of Computer Graphics (2003), pp. 209214.

[WSBW01]

[WTR 03]

[WZ98]

[Yan96]

[Yar67]

[Yee00]

[YPG01]

[ZF03]

75