COMPM076 / GV07 - Introduction to Virtual Environments: Mixed Reality

Transcription

1 COMPM076 / GV07 - Introduction to Virtual Environments: Mixed Reality Simon Julier Department of Computer Science University College London

2 Structure Introduction Display Systems Tracking Systems Display Techniques Interaction Methods 2

3 Introduction Introduction Display Systems Tracking Systems Display Techniques Interaction Methods 3

4 Virtual Environments The focus of this module so far has been on developing virtual environments The user is surrounded by an entirely virtual world Objectives are to do things like induce immersion or presence However,VE is at just one end of the mixed reality continuum 4

5 The Mixed Reality Continuum 5

6 What is Mixed Reality? MR systems combine virtual and real objects 6

7 But This Includes Film Special Effects I, Robot 7

8 What is Mixed Reality? MR systems combine virtual and real objects They run interactively and in real time 8

9 But This Includes Heads Up Displays SportVUE MC1 Wireless Heads Up Display 9

10 What is Mixed Reality? MR systems combine virtual and real objects They run interactively and in real time They register (align) virtual and real objects with one another 10

11 The Ultimate MR System Augmented Reality, a New Way of Seeing, S. K. Feiner, Scientific American, April

12 World-Fixed vs. Camera-Fixed MR When we register with the environment, we have to know where the real world objects are Two types of MR system can be developed: World fixed Camera fixed 12

13 World-Fixed MR Application 13

14 Heads Up Display 14

15 Broadcasting racef/x by sportvision 15

16 Broadcasting 1st & ten by sportvision 16

17 Surveying Courtesy of Mark Francis, Topcon 17

18 Surveying Wide field of view camera Narrow field of view camera Captured image + overlays Topcon GPT-7000i 18

19 Surveying 19

20 Pre-visualisation 20

21 Synthetic Vision for UAVs

22 Navigation 22

23 Navigation Ruggedised pan tilt zoom camera Annotated video shown on bridge 23

24 Navigation 24

25 Advertising 25

26 Advertising 26

27 Location-Aware Information Layar 27

28 Camera-Fixed MR Application 28

29 Lego Digital Box 29

30 MR For Postage US Postal Service Virtual Box Simulator Demo 30

31 Mixed Reality and Games Consoles Eye of Judgement, Sony Computer Entertainment 31

32 Mobile AR Systems Tracking Display Interaction Interface 32

33 Challenges Posed by MR In some ways, MR is easier: Because it is anchored to the real world, tracking and navigation metaphors come for free However, because we are anchoring the graphics to the real world, other challenges arise: The type of display critically affects what can be shown The accuracy of the tracking system critically affects how well the real and virtual objects are registered The way in which the graphics interacts with the real world controls depth perception, etc. 33

34 Summary Mixed reality is a generalisation of VEs to incorporate both real and virtual objects Applications can be world-fixed or camera-fixed They are very widely used, even today The challenges for building good MR systems relate with how the virtual and real content interact 34

35 Display Systems Introduction Display Systems Tracking Systems Display Techniques Interaction Methods 35

36 Building an AR System Display 36

37 Display Systems There are several kinds of display technology which can be used which depend upon the application: Video see-through displays Optical see-through displays Projection displays

38 Video See-Through Display The scene is observed by a camera The graphics are overlaid on the video stream from the camera 38

39 Video See-Through HMDs 39

40 Mobile Phones Given the increase in memory and computational capability, together with handy built in cameras, mobile phones are now a major MR platform 40

41 Tablet-Based PC VisTracker and AR Demo courtesy of InterSense 41

42 Any Content Can be Inserted NRL Virtual Combat Trainer 42

43 For Example UNC Needle-Guided Biopsy Guided maintenance of equipment 43

44 Disadvantages of Video See Through Display Can only see part of real world collected by the camera Narrow field of view Low contrast If the power fails, you can t see anything! Focal length of device is fixed to distance of display Includes both real world and virtual objects at any depth 44

45 Optical See-Through Display Considered the gold standard A user wears a see-through head mounted display The real world and graphics are merged using some kind of optical combiner 45

46 Examples of See-Through HMDs 46

47 Advantages of Optical See-Through Displays The real world can be seen all the time Wide field of view High contrast High colour resolution If the display loses power, you can still see the real world 47

48 Optical See-Through Display 48

49 Optical See-Through Display 49

50 Graphics Can t Block Real World 50

51 Light Leakage is a Significant Problem 51

52 High-Powered Displays Partial Solution 52

53 Projective Displays User wears projector on head that casts images onto scene Reflection is visible 53

54 Projective HMD in Action 54

55 Advantages of Projective Displays Graphics overlaid directly, in terms of depth, with real world The tape can be used to occlude otherwise disruptive objects in the environment 55

56 Disadvantages of Projective Displays If power goes, everything goes Retroreflective tape has to be put everywhere you expect to see virtual content Since intensity falls with the square of distance, only nearby objects can be seen brightly 56

57 Projective MR EasyWeb Show Reel 57

58 Display Technology Summary Three main classes of displays can be used: Video see-through Optical see-through Projective All three have advantages and disadvantges Video see-through is probably the most widely researched and used: Simple Can overwrite any object in the scene 58

59 Tracking Systems Introduction Display Systems Tracking Systems Display Techniques Interaction Methods 59

60 ARToolKit The most widely used system Markers are 2D rectangles with distinctive patterns The pose of the pattern is computed relative to the camera 60

61 Tracking Systems To show the information to the user, we have to know where they are and where they are looking You might think we can get away with just a GPS and compass GPS gives position Compass gives orientation However, the performance can be pretty poor 61

62 GPS Causes Latency MARA, Nokia Nokia 6680 with add-on box containing GPS 62

63 Metal Causes Heading Errors NRL Battlefield Augmented Reality System 63

64 InterSense VisTracker Hybrid vision-inertial system Camera looks for special markers at low speed Inertial system measures velocity and acceleration at high speed Two are combined to give fast, accurate tracking InterSense VisTracker 64

65 Natural Marker-Based Systems Using artificial markers is a pain You have to physically install them on site You have to measure them to work out where they are You have to make sure that you put enough of them in that the tracker can see enough to compute the pose An alternative is to the unmodified environment Model-based Natural marker-based 65

66 Model-Based Tracking The system already knows the structure of all the key features in the environment it can track from The system therefore detects for model features in the frame These are associated with the model The camera pose is computed 66

67 Hybrid Model-Based Tracking System Camera MEMS IMU Interfacing and power! Going Out, G. Reitmayr and T. Drummond, Cambridge University 67

68 Hybrid Model-Based Tracking System Going Out, Gerhard Reitmayr and Tom Drummond, Cambridge University 68

69 SLAM-Based Tracking Developing models of the environment can be at least as difficult as using artificial markers Therefore, an alternative technique is to allow the system to detect and build a map of its own environment This approach is known as Simultaneous Localisation and Mapping (SLAM) 69

70 AR in Small Workspaces 70

71 Parallel Tracking and Mapping Parallel Tracking and Mapping, courtesy of Georg Klein and David Murray, University of Oxford

72 PTAM on an iphone Courtesy of Georg Klein and David Murray, University of Oxford 72

73 AR In Large Workspaces Mapping Large Loops with a Single Hand-Held Camera, L. Clemente and A. Davison and I. Reid and J. Neira and J. Tardos 73

74 Tracking System Summary Reliable and accurate tracking is important Wide-area tracking systems aren t good enough Hybrid and computer vision-based systems are in the works Global coverage is not here yet 74

75 Display Techniques Introduction Display Systems Tracking Systems Display Techniques Interaction Methods 75

76 Building an AR System Interface 76

77 User Interfaces To be of any use, the user has to understand the information which is being shown to them However, there are several issues we have to grapple with, including: Environmental effects Density of information Occlusions Unambiguous labelling 77

78 Annotation and Labelling One common issue is that we want to label the environment However, simple strategies like putting labels at centroids isn t good enough need to look at the screen space of objects 78

79 Labelling The Environment One approach is to use view management Keep track of where the 3D objects project onto the screen 2D annotations are laid out so that: They fit in or near the space of the object They do not interfere with one another The temporal placement of annotations is controlled using a finite state machine

80 Labelling the Environment Work out projection of objects onto display

81 Labelling the Environment Parameterise unoccupied space

82 Labelling the Environment The label location strategy is a finite state machine

83 Labelling the Environment Place annotations into free areas

84 Adaptive Label Management

85 Depth Estimation and Depth Ordering We often want to know the order or range of virtual objects relative to the environment Stereoscopic cues are of limited use Secondary cues tend to bias virtual object further away than real objects Various supplementary cues can be provided to convey depth information

86 Viewing Occluded Objects Threat Location Threat location User 86

87 X-Ray Vision Through Display Styles 87

88 Before 88

89 After 89

90 Developed Adaptation to Ambient Conditions Color of text is function of background over which it is drawn Algorithms identified by evaluation study at VaTech Therefore, the colour of the augmentation has to change based on scene composition Camera views scene Color and intensity processed to change label colors Camera

91 Developed Adaptation to Ambient Conditions Label ( Parking Lot ) changes color to contrast with background

92 Registration Errors Can Be Confusing 92

93 Reduce Effects by Adapting the Display Increasing registration error 93

94 Bounding Regions for Objects y(k)= [ŷ(k), Y(k)] P M(k) = f(m 1 (x 1 (k)),,m t (x t (k))) x i (k)=[x ^ i (k), X i (k)] p p Screen Head World For each vertex in object: Project each vertex through projection equations to work out pixel position Calculate mean and 1σ covariance ellipse due to registration errors Approximate ellipses by set of points Calculate bounding regions defined by these ellipses

95 Registration Error Adaptation Target window Target designated 95

96 Registration Error Adaptation Error bounds Target window Target registration error region overlaps distracters 96

97 Registration Error Adaptation Calculate overlap of target and distracter error regions 97

98 Registration Error Adaptation targetobject is in the middle Add aggregate representation and text message 98

99 Dealing With Information Overload 99

100 Lots of Data is Extremely Confusing 100

101 Results 101

102 Interaction Methods Introduction Display Systems Tracking Systems Display Techniques Interaction Methods 102

103 Building an AR System Interaction Interface 103

104 Interaction 104

105 Heads Up Interaction What s that object? 105

106 Selection and Interaction 106

107 Multimodal Integrator Architecture Speech Recogniser Gesture Recogniser Weighted list of sentences Weighted list of selections Multimodal Integrator Object with highest probability 107

108 Example Multi-modal Interaction 108 Probabilistic Algorithms, Integration, and Empirical Evaluation for Disambiguating Multiple Selections in Frustum-Based Pointing, G. Schmidt et al.

109 Summary Many interaction challenges are the same as those in VE and the same techniques can be used Multi-modal interaction is one possibility which combined speech and gesture to support intuitive interaction 109