Light transport matrices

Transcription

1 Light transport matrices , , Computational Photography Fall 2018, Lecture 18

2 Course announcements Homework 5 has been posted. - Due on Friday November 9 th. Any problems with homework 4? No elevator pitch presentations for final projects. Extra office hours this week: - Monday 1:30-3:30 pm. - Tuesday noon-2:00 pm. - Friday s office hours will be held by Alankar. Great talk this Thursday: Eric Fossum, inventor of the CMOS sensor, will talk about quantum (i.e., photon-counting) CMOS sensors.

3 Overview of today s lecture Leftover from last time: Generalized bas-relief ambiguity. The light transport matrix. Image-based relighting. Photometric stereo revisited. Optical computing using the light transport matrix. Dual photography.

4 Slide credits These slides were directly adapted from: Matt O Toole (CMU).

5 The light transport matrix

6 How do these three images relate to each other?

7 How do these three images relate to each other?

8

9

10 why is the error not exactly zero?

11

12

13

14

15

16

17

18

19

20

21

22

23

24 light transport matrix Can you think of a case where we have a very large m?

25 Use a projector What does each row and column of T correspond to here?

26 Use a projector How would you go about measuring the light transport matrix?

27 Image-based relighting

28 Let s say I have measured T. What does it mean to right-multiply it with some vector l?

29 Image-based relighting: Use the measurements I already have of the scene (the pictures I took when measuring T) to simulate new illuminations of the scene.

30

31 Great demonstration:

32

33 Photometric stereo revisited

34

35 What is this?

36

37

38 v=4gilaotjhno

39 Optical computing using the light transport matrix

40

41

42

43

44

45

46

47

48 What do we call these patterns?

49

50

51

52

53

54

55

56

57

58

59

60 How would you measure the light transport matrix T?

61 How would you measure the light transport matrix T? Exhaustive/naïve approach: turn on projector pixels one at a time and take a photo for each of them. What does each photo correspond to in T?

62 How would you measure the light transport matrix T? Exhaustive/naïve approach: turn on projector pixels one at a time and take a photo for each of them. How many photos do we need to capture?

63

64 Inverse transport

65

66

67

68

69 How do you solve this problem if you know the light transport matrix T?

70 What do we do here?

71 What if T is not invertible?

72 How do you usually solve for l when T is large?

73 Reminder from lecture 10: Gradient descent Given the loss function: E f = Gf v 2 Minimize by iteratively computing: f i+1 = f i η i r i, r i = v Af i, η i = ri T r i r i T Ar i for i = 0, 1,, N Is this cheaper than the pseudo-inverse approach? We never need to compute A, only its products with vectors f, r. Vectors f, r are images. Because A is the Laplacian matrix, these matrix-vector products can be efficiently computed using convolutions with the Laplacian kernel.

74 Gradient descent in this case Given the loss function: E f = Gf v 2 Minimize by iteratively computing: f i+1 = f i η i r i, r i = v Af i, η i = ri T r i r i T Ar i for i = 0, 1,, N Is this cheaper than the pseudo-inverse approach? We never need to compute A, only its products with vectors f, r. Vectors f, r are images. What are f, r in this case? Because A is the Laplacian matrix, these matrix-vector products can be efficiently computed using convolutions with the Laplacian kernel. How do we compute matrix-vector products efficiently in this case?

75

76

77 Dual photography

78 How do the light transport matrices for these two scenes relate to each other?

79 Helmholtz reciprocity: The two matrices are the transpose of each other. Great demonstration:

80 References Basic reading: Sloan et al., Precomputed radiance transfer for real-time rendering in dynamic, lowfrequency lighting environments, SIGGRAPH Ng et al., All-frequency shadows using non-linear wavelet lighting approximation, SIGGRAPH Seitz et al., A theory of inverse light transport, ICCV These three papers all discuss the concept of light transport matrix in detail. Debevec et al., Acquiring the reflectance field of a human face, SIGGRAPH The paper on image-based relighting. Woodham et al., Photometric stereo: A reflectance map technique for determining surface orientation from image intensity, IUSIA The original photometric stereo paper. O Toole and Kutulakos, Optical computing for fast light transport analysis, SIGGRAPH Asia The paper on eigenanalysis and optical computing using light transport matrices. Sen et al., Dual photography, SIGGRAPH The dual photography paper. Additional reading: Peers et al., Compressive light transport sensing, TOG Wang et al., Kernel Nyström method for light transport, SIGGRAPH These two papers discuss alternative ways for efficient acquisition of the light transport matrix, using assumptions on its algebraic structure.