a real convergence computer vision finally becoming useful

Transcription

1 Introduce the field a real convergence computer vision finally becoming useful Pikachu Nintendo - HoloLens Microsoft - 1

2 Niantic Ingress Pokemon Go Nintendo, Niantic Image Independent Logo Nintendo - 2

3 Snapchat Photo

4 Images Microsoft Like magic! Where did that come from?! Even a company building a competitor called MagicLeap that has put out a few screen recordings. 4

5 5

6 6

7 Top: Bregler et al. Bottom Beowolf production shots 7

8 Furukawa et al. 8

9 [Bokeloh et el. EUROGRAPHICS 2009] 9

10 [Ma et al. EGSR 2007] 10

11 Explain the diagram. So what happens when the estimation of these models is not useful when they are incorrect or imprecise or insufficient? 11

12 Question audience what is wrong here? Video - Playing Pokémon GO in New York - Tech Insider 12

13 Image - What went wrong here? 13

14 Image - What went wrong here? 14

15 Image - Question audience what is wrong here? Next slide: but when you get it right, it can be transformative. 15

16 Magic Leap video - Where else are these models useful? Half face shot is next. 16

17 Beyond augmented reality to VFX Can anyone guess what is going on in this image? What artifact do you see? 17

18 Ex Machina VFX Showreel Double Negative - A young programmer is selected to participate in a ground-breaking experiment in synthetic intelligence by evaluating the human qualities of a breath-taking humanoid A.I. Academy Award for best visual effects, How did they fill in the bits _behind_ the newly transparent parts? 18

19 19

20 Image Jason Corso

21 21

22 OK, so before I give you a little introduction about myself, who do we have in the room? Undergrads? Masters? PhDs? And what s your experience in visual computing? Who has taken a graphics course? Vision course? Machine learning? Interaction? 22

23 I ve been working in visual computing research now for 10 years, and in that time I ve combined state-of-the-art computer vision techniques to create new interactive graphics applications. 23

24 Well, he s one example result. The user provides start and end pair of images, and the system generates an exploration of the video collection that takes in those views. This is computed completely automatically, just from the visual information. In this result, the system smoothly and automatically moves from looking at Big Ben to looking at the London Eye in this composite motion made from 4 videos, the camera moves from one side of the river to the other and rotates 180 degrees to take in both views. Tompkin et al. SIGGRAPH 2012 Videoscapes 24

25 Here s what it looks like in motion. When we move through a video s timeline, the video foci moves spatially in the context. We can see the comings and goings of the people in the scene, across different times potentially across many months. In this way, by embedding the videos within a shared context, we can move away from the typical linear sequential presentation of videos, to build new interactions to compare and contrast spatio-temporal events. FOCUS + CONTEXT -> great for YURT resolution demo Tompkin et al. UIST 2013 Vidicontexts 25

26 Tompkin et al. UIST 2013 Vidicontexts 26

27 How might we paint in free space within a volume, with a hand-held pen, and have those paintings appear exactly where you placed them? Tompkin et al. UIST D light field pen sensing 27

28 Now, in my time as a masters and doctoral student, many things were hard for me. 28

29 Especially tough right at the beginning of your studies. 29

30 30

31 Especially tough right at the beginning of your studies. 31

32 32

33 One person is assigned at random at the beginning of the seminar to lead the discussion. Everybody leads discussion at least once in the course. The discussion leader receives a digest of the submitted questions just before the seminar. The discussion leader raises questions appropriately throughout the discussion, covers future work aspects, and finally provides a summary of the strengths and weaknesses of the techniques and of the discipline. 33

34 34

35 35

36 36

37 37

38 38

39 You spend a lot of time in high school and later on in college, under certain disciplines, learning to write. But learning to read efficiently when we move to reading academic texts is an important skill for your dissertations or thesis, and it s not really taught as much. 39

40 40

41 PatchMatch + Global Patch Collider or TreeCANN Image Barnes et al., PatchMatch, SIGGRAPH

42 Stereo diagram MATLAB - Model column architecture Goesele et al. Multi-view Stereo Revisited

43 Learning to be a depth camera Project Tango Venture Beat

44 Dou et al., SIGGRAPH

45 Left Shotton et al., Real-Time Human Pose Recognition in Parts from a Single Depth Image - Right Gall et al., Motion Capture Using Joint Skeleton Tracking and Surface Estimation 45

46 Debevec et al. Rendering Synthetic Objects into Real Scenes (SIGGRAPH 1998) NOTE: This is _not_ a difficult illumination estimation paper, but it is one. Uses HDR images of environment maps to light the virtual world. We will read papers on more advanced problems. 46

47 Image Song et al Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction 47

48 Video WIRED - Has anyone heard of Magic Leap? $800 million seed round WIRED 48

49 Image Wikipedia

50 A great literature review 50

51 Video Märki et al

52 Video Märki et al

53 Images MIT intrinsic image database Roger Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman, Ground truth dataset and baseline evaluations for intrinsic image algorithms, in Proceedings of the International Conference on Computer Vision (ICCV),

54 Video Laffont et al., Coherent Intrinsic Images from Photo Collections, 54

55 Video Face2Face: Real-time Face Capture and Reenactment of RGB Videos - Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner CVPR

56 56

57 57

58 58

59 59

60 60

61 61