Communication in Virtual Environments. Communication in Virtual Environments

Transcription

1 Communication in Virtual Environments Igor Linköping University Outline Goals of the workshop Virtual Environments and related topics Networked Collaborative Virtual Environments Virtual Environments on the Web Virtual Humans 7/25/01 CUC01 Igor S. Pandzic 2 1

2 Goals of the workshop Give a basic understanding of Virtual Environments (VE) and related topics Present technologies for using VE in communications Discuss applications of presented technologies 7/25/01 CUC01 Igor S. Pandzic 3 Virtual Environments and related topics Classification of terms Virtual Environment basics Virtual scene Rendering 7/25/01 CUC01 Igor S. Pandzic 4 2

3 Classification of terms 2D 3D Computer Animation Real-time graphics Virtual Worlds Virtual Humans Special effects CAD/CAM Virtual Environments 7/25/01 CUC01 Igor S. Pandzic 5 Computer Graphics 2D graphics GUI PowerPoint PhotoShop 3D graphics Jurassic Parc, Toy Story etc. AutoCAD Difference is in internal data representation! 7/25/01 CUC01 Igor S. Pandzic 6 3

4 2D Computer Graphics 3D Non-Real Time, off-line Real Time, Interactive Image-per-image production High quality Complex simulations possible At least 10 images/sec Less quality Interaction with 3D objects 7/25/01 CUC01 Igor S. Pandzic 7 2D Special effects Computer Graphics 3D Non-Real Time, off-line Computer Animation Real Time, Interactive Everything is possible, result must be perfect Increasng use of computers Film production Ads, cartoons, short films, even big movies 7/25/01 CUC01 Igor S. Pandzic 8 4

5 2D Computer Graphics 3D Interactive simulation of places and objects non-real Time, off-line Real Time, Interactive Special effects Replace Computer our Animation senses by computer outputs Plug our actions into computer inputs Huge variety of special VR hardware, sometimes strange-looking Virtual Environments/Worlds Virtual Reality 7/25/01 CUC01 Igor S. Pandzic 9 Special effects 2D Non-Real Time, off-line) Computer Animation CAD Computer Graphics 3D Virtual Humans Real Time, Simulation of human beings on a computer Interactive Applications in entertainment, Virtual manufacturing, communications... Environments/Worlds Virtual Reality 7/25/01 CUC01 Igor S. Pandzic 10 5

6 Special effects Virtual Environments on the Web Emerging Non-Real technologies, Time, competing off-linestandards 2D Networked Computer Animation Collaborative Virtual Environments Computer Graphics 3D Real Time, Interactive Virtual Environments/Worlds Web3D Multiple users sharing Virtual a Humans VE through the network Virtual Reality CAD NCVE 7/25/01 CUC01 Igor S. Pandzic 11 Special effects 2D Non-Real Time, off-line Computer Animation CAD Computer Graphics 3D Virtual Humans Real Time, Interactive Virtual Environments/Worlds Web3D Virtual Reality NCVE 7/25/01 CUC01 Igor S. Pandzic 12 6

7 Digression: The Utah Teapot Developped in 1975 at University of Utah (Newell) Traditionally used as test object for Computer Graphics algorithms It became a symbol of CG 7/25/01 CUC01 Igor S. Pandzic 13 Virtual Environment basics Virtual Scene Components of a scene Rendering Getting finished images of a virtual scene 7/25/01 CUC01 Igor S. Pandzic 14 7

8 Photography scene Light Objects Materials Photo Camera 7/25/01 CUC01 Igor S. Pandzic 15 Virtual scene Virtual Light Virtual Virtual Objects Materials VirtualCamera Real Photo 7/25/01 CUC01 Igor S. Pandzic 16 8

9 Definition of virtual Everything that is virtual exists only in computer memory This memory is digital, posing two problems: How to represent natural phenomena (objects, lights, etc.) digitally? How to get a nice image out of it in the end? 7/25/01 CUC01 Igor S. Pandzic 17 Rendering Virtual Light Virtual Virtual Objects Materials Real Photo Virtual Camera 7/25/01 CUC01 Igor S. Pandzic 18 9

10 Networked Collaborative Virtual Environments Basic concept Technical challenges Example NCVE systems 7/25/01 CUC01 Igor S. Pandzic 19 NCVE - basic concept Geographically distant users interact in a shared Virtual Environment 7/25/01 CUC01 Igor S. Pandzic 20 10

11 How does it work? VERY simplified explanation: HOST 1 1 HOST NETWORK HOST 2 HOST 4 7/25/01 CUC01 Igor S. Pandzic 21 NCVE technical challenges Scaleability Network topologies Space structuring Real time simulation User representation Human communication 7/25/01 CUC01 Igor S. Pandzic 22 11

12 Scaleability Problem: How to support increasing numbers of users? Solutions: Clever network topology Space structuring Area Of Interest Management (AOIM) Level Of Detail (LOD) management 7/25/01 CUC01 Igor S. Pandzic 23 Network topologies Problem: Efficient message transfer AOIM support HOST 1 Session management World persistence Solutions: Peer-to-peer topology HOST 2 Multicast topology Client/server topology Multiple servers topology 1 2 NETWORK HOST 3 HOST 4 7/25/01 CUC01 Igor S. Pandzic

13 Space structuring Problem: Divide the VE into manageable units with respect to: number of users world data base size coordinates precision Solutions: Separate servers Uniform geometrical structure Free geometrical structure 7/25/01 CUC01 Igor S. Pandzic 25 Real time 3D simulation Problem: Display the VE at interactive frame rates Solution: Use a real time toolkit including: LOD management Hierarchical scene graph management Parallel scene graph traversal Good integration with hardware 7/25/01 CUC01 Igor S. Pandzic 26 13

14 User representation Problem: Let the users perceive each other Solutions: Simple cube-like appearances Non-articulated characters Articulated rigid-segment body Deformable body and face 7/25/01 CUC01 Igor S. Pandzic 27 Human communication Problem: Support life-like communication in NCVE Solutions: Text Audio Gestures Facial expressions 7/25/01 CUC01 Igor S. Pandzic 28 14

15 NCVE systems NPSNET DIVE SPLINE VLNET Blaxxun Doom Adobe Atmosphere 7/25/01 CUC01 Igor S. Pandzic 29 Potential applications Virtual teleconferencing Collaborative work involving 3D design Multi-user game environments ecommerce Medical applications Distance learning/training 7/25/01 CUC01 Igor S. Pandzic 30 15

16 Virtual Environments on the Web Why VE on the Web? Possible solutions for VE on the Web: VRML / X3D MPEG-4 Java3D Java applet solutions Plug-in solutions Substitute solutions ( false 3D ) 7/25/01 CUC01 Igor S. Pandzic 31 Why VE on the Web Enable 3D content on the Web Better visualization in many fields Enable new applications 7/25/01 CUC01 Igor S. Pandzic 32 16

17 VRML Virtual Reality Modeling Language ISO Standard since (VRML97) Syntax for describing virtual scenes Scene Graph concept - objects placed in a tree Animation with sensors, interpolators Programming with scripts or external applications 7/25/01 CUC01 Igor S. Pandzic 33 VRML Most important standard format for 3D data Problems: ASCII, heavy, needs encoding (solutions exist, but not standard) not (yet) universally available in browsers - plug-in required New version in development 7/25/01 CUC01 Igor S. Pandzic 34 17

18 VRML 200X International Standard ISO/IEC 14772:200X Next generation of VRML Improve functionality (new nodes, external interfaces, object model, profiles, web integration) Improve encoding (XML and binary versions) 7/25/01 CUC01 Igor S. Pandzic 35 MPEG-4 International Standard ISO/IEC IS status 1999 for version 1), 2000 version 2 Huge industrial participation several hundreds of experts involved estimated $500M spent so far Includes very extensive tools for audio, video and 3D object coding 7/25/01 CUC01 Igor S. Pandzic 36 18

19 MPEG-4: New paradigm for AV coding Set of dynamic AV objects Composition on decoder Interaction Synthetic objects 7/25/01 CUC01 Igor S. Pandzic 37 3D in MPEG-4 Based on VRML Efficient encoding: BInary Format for Scenes (BIFS) Data streaming, animation streaming Special nodes Face and Body Animation Animated mesh 7/25/01 CUC01 Igor S. Pandzic 38 19

20 Java3D Java API for 3D graphics programming high-level API Problems performance? Not a core part of Java, difficult to install in a browser 7/25/01 CUC01 Igor S. Pandzic 39 Java applet solutions Applets capable of rendering interactive 3D graphics (usually VRML) Avoid need for plug-in, therefore the most practical solution at the moment Problems: size of the applet Some examples: 7/25/01 CUC01 Igor S. Pandzic 40 20

21 Plug-in solutions 3D renderer in a browser plug-in Problems: No widespread standard (yet!) Need to download and install Some examples: /25/01 CUC01 Igor S. Pandzic 41 Substitute solutions ( false 3D ) Use other techniques (usually image based) to achieve a semblance of 3D Panoramic images Animations Examples Quicktime VR: (like Quicktime, but better) Talking Virtual Characters: 7/25/01 CUC01 Igor S. Pandzic 42 21

22 Future of VE on the Web Technical challenges to bring Virtual Environments to the Web are being solved Lots of interest in industry What will be the killer application? 7/25/01 CUC01 Igor S. Pandzic 43 Virtual Humans Virtual Humans Basic concepts Representation Animation Behaviour Virtual Humans in NCVEs Standards for Virtual Humans Virtual Humans on the Web Applications for Virtual Humans 7/25/01 CUC01 Igor S. Pandzic 44 22

23 Virtual Humans Simulation of human beings on computers Representation Graphical Functional Animation Behavior 7/25/01 CUC01 Igor S. Pandzic 45 Graphical representation To define the shape and color properties of the virtual human Skin Clothes Hair Material properties 7/25/01 CUC01 Igor S. Pandzic 46 23

24 Functional representation The data structures necessary for the animation of the virtual human Skeleton Necessary for body animation Usually simplified Facial structure Necessary for facial animation Usually contains data on facial regions or characteristic points (lips, eyes, eyebrows etc.) 7/25/01 CUC01 Igor S. Pandzic 47 Functional and graphical representation must be linked - this usually means some more data 7/25/01 CUC01 Igor S. Pandzic 48 24

25 Animation Body animation Skin deformation Facial animation Clothes animation Hair animation 7/25/01 CUC01 Igor S. Pandzic 49 Body animation Body is animated by animating the skeleton A good approximation of anatomycal skeleton: 75 Degrees of Freedom + 2x30 DOF for hands, mostly rotational Real skeleton has many more, including translation Rougher approximation often used r_wrist r_elbow r_hand_center r_scapula r_shoulder r_clavicle r_cp_pelvic r_hip r_knee vt6 head_top vl2 pelvis vt1 l_cp_pelvic l_hip l_knee r_ankle l_ankle r_subtalar l_subtalar r_mid_foot l_mid_foot r_toe l_toe Skeleton with 75 DOF vc8 vc7 vt5 head vt4 vl3 l_scapula l_shoulder l_clavicle l_elbow l_wrist l_hand_center Hand skeleton with 30 DOF 7/25/01 CUC01 Igor S. Pandzic 50 25

26 Body Animation Complex because of many degrees of freedom Use of inverse kinematics Optical or magnetic body tracking often used to obtain real movement data 7/25/01 CUC01 Igor S. Pandzic 51 Skin deformation Based on skeleton movement, skin deforms Relatively complex, often ignored (body composed of rigid segments) 7/25/01 CUC01 Igor S. Pandzic 52 26

27 Facial animation Face: extremely important communication channel Even slightest facial expressions meaningful Extraordinary complexity of bone and muscle structure 7/25/01 CUC01 Igor S. Pandzic 53 Facial Animation More complex simulation systems try to approximate muscle dynamics, others concentrate on geometrical effect on the skin FA system is usually controlled by a set of parametres activating facial actions 7/25/01 CUC01 Igor S. Pandzic 54 27

28 Facial Animation Some FA systems work only with very high-level parameters: text to pronounce and emotions to display 7/25/01 CUC01 Igor S. Pandzic 55 Clothes animation Most often, clothes are represented as colour on the body geometry - no separate layers for skin and clothes More sophisticated models simulate clothes physically 7/25/01 CUC01 Igor S. Pandzic 56 28

29 Hair animation Most often ignored (rigid hair model) Models for hair animation: encouraging results, but very computing-intensive Successful fur simulation seen in movies 7/25/01 CUC01 Igor S. Pandzic 57 Behaviour Simple behaviors can be achieved Walking, walking on trajectory Obstacle avoidance Grasping objects High level autonomous behaviour (research) Decision-making Interaction with users 7/25/01 CUC01 Igor S. Pandzic 58 29

30 Reasons for Virtual Humans in NCVE Requirements on user representation: Perception Localization Identification Visualization of interest focus Visualization of actions Communication Can a simple representation (e.g. a cube) meet the requirements? 7/25/01 CUC01 Igor S. Pandzic 59 Standards for Virtual Humans MPEG-4 Face and Body Animation (FBA) VRML H-ANIM 7/25/01 CUC01 Igor S. Pandzic 60 30

31 Virtual Humans on the Web Very new area Potential usage as virtual hosts, presenters, salespersons, newscasters etc. Technology: images video streaming Java applets 3D plug-ins 7/25/01 CUC01 Igor S. Pandzic 61 Virtual Humans - Applications Film and TV Ergonomy studies (vehicles, furniture etc.) Work process efficiency studies Clothing industry Games Telecommunication Medical simulations Web, ecommerce 7/25/01 CUC01 Igor S. Pandzic 62 31

32 Further reading Further reading 3D Computer Graphics, Alan Watt, Addison-Wesley Introduction to Computer Graphics, Foley, Van Dam, Hughes, Phillips, Addison-Wesley Avatars in Networked Virtual Environments, Capin, Pandzic, Magnenat-Thalmann, Thalmann, John Wiley & Sons Networked Virtual Environments: Design and Implementation, Singhal, Zyda, Addison-Wesley Computer Facial Animation, Parke, Waters, A K Peters Ltd. Vrml 2.0 Sourcebook, Ames, Nadeau, Moreland, John Wiley & Sons Web3D consortium: MPEG-4: Java3D: 7/25/01 CUC01 Igor S. Pandzic 63 32