Computer Graphics Presenting The Visual Future

Transcription

1 Computer Graphics Presenting The Visual Future Boris Ajdin Max Planck Institut für Informatik

2 Introduction What is computer graphics? Standard view of the field Sci-Fi like applications Multi-disciplinary computer graphics Combining other sciences with graphics Combining graphics with other sciences Computational future Using NVIDIA CUDA platform

3 Computer Graphics Wikipedia definition: Computer graphics broadly studies the manipulation of visual and geometric information using computational techniques. Computer graphics as an academic discipline focuses on the mathematical and computational foundations of image generation and processing rather than purely aesthetic issues. TU Berlin definition: Computer Graphics is about digital models for three-dimensional geometric objects as well as images. Answers.com definition: A branch of computer science that deals with the theory and techniques of computer image synthesis.

4

5

6 CG standard stuff

7 CG recent standard stuff Movies Lord of the Rings Trilogy, Transformers, The Matrix,... Games Unreal Tournament 3, BioShock, Crysis,... OS GUI Windows Vista, Mac OS X, KDE, Gnome,...

8 CG recent standard stuff Movies Lord of the Rings Trilogy, Transformers, The Matrix,... Games Unreal Tournament 3, BioShock, Crysis,... OS GUI Windows Vista, Mac OS X, KDE, Gnome,...

9 CG recent standard stuff Movies Lord of the Rings Trilogy, Transformers, The Matrix,... Games Unreal Tournament 3, BioShock, Crysis,... OS GUI Windows Vista, Mac OS X, KDE, Gnome,...

10 CG recent standard stuff Movies Lord of the Rings Trilogy, Transformers, The Matrix,... Games Unreal Tournament 3, BioShock, Crysis,... OS GUI Windows Vista, Mac OS X, KDE, Gnome

11 CG recent standard stuff Movies Lord of the Rings Trilogy, Transformers, The Matrix,... Games Unreal Tournament 3, BioShock, Crysis,... OS GUI Windows Vista, Mac OS X, KDE, Gnome,...

12 CG recent standard stuff Movies Lord of the Rings Trilogy, Transformers, The Matrix,... Games Unreal Tournament 3, BioShock, Crysis,... OS GUI Windows Vista, Mac OS X, KDE, Gnome,... This is now - can we create the future today?

13 Desired high-tech Photo-realistic rendering Fast image based rendering (IBR) systems Super-resolution cameras and projectors 3D capture and display devices Intuitive and powerful user interfaces A (possibly) distant (possible) future: True virtual reality Computer-brain interfaces Real life smart machine-environment interaction...

14 Content-aware resizing Various image/video formats: 4:3, 5:4, 16:9,... Annoying black spaces or cropping on displays with different format 4:3 image 4:3 stretched 4:3 cropped 16:9 image

15 Seam Carving for Content-Aware Image Resizing, Shai Avidan and Ariel Shamir, Siggraph Insert or remove a seam (connected path along one dimension), with the lowest energy.

16 Minimize the seam cost, n c given the energy function s =min E s =min e I si s s i=1 e (for vertical seams) Use dynamic programming to find the minimum seam: compute cumulative cost, starting from the second row, and then backtrack Example energy function I I x y e HoG I = max HoG I x, y

17

18

19

20 Completing images What if you want to: Remove somebody or something from an image? Restore damaged photographs? Options we have at hand: Hand work in some photo-editing tool, e.g. Photoshop Smart software solution

21 Image Completion Using Global Optimization, Nikos Komodakis and Georgios Tziritas, CVPR 2006 Joint framework for image completion and texture synthesis

22 Copy the missing information from some other region of the image Define source nodes and destination nodes in the image Use Markov Random Fields to minimize the resulting image energy V i l = [ p 2 ][ w w h h,, N ] M ni p I 0 ni p I 0 l p E l i = V i l i i=1 i, j V i, j l i, l j

23

24 Depth image camera Next big step: a one click 3D depth camera one depth value for each pixel position Desert of the real: 3D depth with multiple clicks using an ordinary camera! Even today this approach could be applied to a single camera device, but: Either the problem of cost and bulkiness Or not usable for dynamic scenes

25 Confocal Stereo, Samuel W. Hasinoff and Kiriakos N. Kutulakos, ECCV 2006 The principle of confocal consistency If the input radiance is constant then: With varying camera aperture, the intensity of an in-focus pixel varies in a scene independent way Hence, it can be precomputed with radiometric camera calibration Given the aperture, focus and confocal consistency coefficient we can compute depth

26

27

28 Novel UI systems Standard one device one click systems do not provide enough usability and power for the user (mouse, light pen,...) Computer keyboard is multi-touch, but is not graphic-oriented Hot topic multi-touch GUIs Apple iphone, Microsoft Table PC,...

29 Using FTIR principle: Infra red LED are placed on the edges of the acrylic pane We back-project the desired graphical content An IR camera, using FTIR, detects contact points with the screen Jeff Han

30 Espionage Sci-Fi movies: super innovative hackers and useless system-admins (password: GOD :) ) Sci-Fi movies (cont): computer screens reflect on users face (human skin is diffuse?!?!) Reality: environment from reflecting (shiny) objects Scenario: Dimly lit room Bright enough computer screen Reflecting object

31 Corneal Imaging System: Environment from Eyes, Ko Nishino and Shree K. Nayar, IJCV 2006 Fit a model of human eyeball to the recorded high-res image Localize the cornea Recover the reflected image from the eye

32

33

34

35

36 Light/reflectance fields Image based rendering - IBR: use photographs (as photo-realistic as it gets) to create arbitrary renderings If you want to enable novel views of an object we are talking about light fields (4D functions) Static lighting If you want to enable relighting then we have 4D reflectance field Only one view-point Relightable & different viewing directions <=> 8D reflectance fields

37 Problems with these fields: Acquisition time too long: Images from different position Varying lighting Machine controlled acquisition process required: Robotic arm camera/light control Turntables Storage costs Often > images Principle problem Not clear how to capture concave objects for relighting ( Holly Grail of IBR)!!!

38 Light fields

39

40 How 'bout 3D? Displaying 3D content Holograms Red/blue images Using polarized light VR glasses,... Usual problems Number of George Washington notes required Lack of color Non-interactive content Self-occlusion not properly addressed...

41 Rendering for an Interactive 360 Light Field Display, Andrew Jones at al, Siggraph Nuts & bolts needed: A spinning mirror (45 tilt angle) A motion control motor (for spinning) Anisotropic holographic diffuser A fast DLP projector (alternating between red, green and blue colors) A PC

42 Use a recorded light field For each mirror position compute and project the closest light field view Correct vertical parallax effect

43 Voilà 3D Display

44 NPR Photo-realistic rendering is one of CG Holly Grails Often the desired visual effects are achieved by using wrong physics of object-light interactions: cartoons, sketches, art, etc Can we create automated non-photo realistic renderings (NPR)? Yes we can: NPAR, Sin City

45 NVIDIA CUDA Moore's law processor power increases by a factor of 2 every ~18 months. Typical CPU 2 CISC cores, ~2 GHz Ideal machine supercomputer: IBM Blue Gene/P 1PFLOPS processors Lots of $ (initial cost, cost of space, power, cooling) Not for everyone.

46 GPU Power Greatly overlooked until recently Now provides unified shader architecture

47 >128 FPUs, ~80 GB/s memory I/O

48 Optimization idea: Isolate relatively simple operations performed on multiple data Store all data on the GPU (internal GPU bandwidth = ~20x GPU-CPU bandwidth) Perform as many operations as possible in parallel. Stream the final results back to the main memory

49 Total balance: Initial costs: a PC + GeForce 8 class GPU = ~1000$ Learning curve: basics of parallel computation Implementation cost: CUDA provides a simple C interface for initializing GPU memory management and computations Hence soon a TFLOPS for everyone!

50 Matrix multiplication P = M * N (size w x w) Standard idea: One thread for each element in P Too much memory copying, not enough parallelism

51 Highly-parallel idea: Each thread computes one block in P (with more then one element) Less memory overhead More efficient use of the available processing power

52 Full code for (int a = abegin, b = bbegin; a <= aend; a += astep, b += bstep) { // Declaration of the shared memory array As used to // store the sub-matrix of A shared float As[BLOCK_SIZE][BLOCK_SIZE]; // Declaration of the shared memory array Bs used to // store the sub-matrix of B shared float Bs[BLOCK_SIZE][BLOCK_SIZE]; // Load the matrices from device memory to shared // memory; each thread loads one element of each matrix AS(ty, tx) = A[a + wa * ty + tx]; BS(ty, tx) = B[b + wb * ty + tx]; syncthreads(); // to make sure the matrices are loaded // Multiply the two matrices together; each thread // computes one element of the block sub-matrix for (int k = 0; k < BLOCK_SIZE; ++k) Csub += AS(ty, k) * BS(k, tx); // Make sure that the preceding computation is done // before loading two new sub-matrices of A and B syncthreads(); } // Write the block sub-matrix to device memory; // each thread writes one element int c = wb * BLOCK_SIZE * by + BLOCK_SIZE * bx; C[c + wb * ty + tx] = Csub; global void matrixmul( float* C, float* A, float* B, int wa, int wb) { int bx = blockidx.x; int by = blockidx.y; //Block index int tx = threadidx.x; int ty = threadidx.y; // Thread index // Index of the first sub-matrix of A processed by the block int abegin = wa * BLOCK_SIZE * by; // Index of the last sub-matrix of A processed by the block int aend = abegin + wa - 1; // Step size used to iterate through the sub-matrices of A int astep = BLOCK_SIZE; // Index of the first sub-matrix of B processed by the block int bbegin = BLOCK_SIZE * bx; // Step size used to iterate through the sub-matrices of B int bstep = BLOCK_SIZE * wb; // Csub is used to store the element of the block sub-matrix // that is computed by the thread float Csub = 0; // Loop over all the sub-matrices of A and B // required to compute the block sub-matrix }

53 CG jobs Prosperous future in CG: Games Movies Commercials Graphical/3D design Computer generated art Robotics&vision...

54 Big giant heads of CG Check out these names on the web: Marc Levoy Pat Hanrahan Paul Debevec Phillip Slusallek Fredo Durand Ramesh Raskar Hans-Peter Seidel Michael Cohen Ravi Ramamoorthi Shree Nayar

55 .