Appearance Editing: Compensation Compliant Alterations to Physical Objects

Transcription

1 Editing: Compensation Compliant Alterations to Physical Objects Daniel G. Aliaga Associate Professor of Purdue University

2 <slides removed>

3 Presentation Outline Previous work editing overview Formulations of compensation compliancy Solutions to improving/ensuring compliancy Additional tools to improve appearance editing Concluding thoughts and future work 29

4 Notation q i : surface point i n i : surface normal at q i A s (q i ): idealized surface albedo at q i under maximal projector illumination p j : projector j T(q i ): target appearance color at q i 30

5 Projector Light Radiance Availability Gamut of achievable colors depends on available light radiance at surface point q i Projector maximum illumination I max Projector-surface point visibility v ij Projector-surface point distance d ij Projector-surface point orientation (p j q i, n i ) Surface point albedo A s (q i ) L p q i = A s (q i ) P j=1 p j q i I max v ij n i p j q i 2 p j q i projector (p j, I max ) visible? (v ij ) d ij n i surface point (q i, A s q i ) 30

6 Method 1: Luminance Compliancy Simplest way to model compliancy is with a scalar value Luminance estimated by RGB color space intensity: I c (c R + c G + c[b])/3, or I c c R 2 + c G 2 + c B 2 Compensation compliant if the intensity of the target color at q i is less than or equal to the intensity of the available light radiance: I T q i I L p (q i ) I T(q i ) I L p (q i ) T(q i ) L p (q i ) Inaccuracy: does not model color s chrominance 31

7 Method 2: RGB Color Space Compliancy RGB channels encapsulate both a color s chrominance and luminance (in some intertwined way) Compensation compliant if each channel s target intensity is less than or equal the per channel available light radiance: T q i [R] L p q i [R] T q i [G] L p q i [G] T q i B L p q i B L p (q i )[G] T(q i )[R] T(q i )[B] L p (q i )[R] L p (q i )[B] T(q i )[G] T(q i ) L p (q i ) Inaccuracy: perceived chrominance not constant in each channel mixed in some intertwined way 32

8 Method 3: CIELAB Color Space Compliancy Accurate modeling of compensation compliancy must be able to precisely measure intensity at any given spectral wavelength or chrominance Lab space dissociates chrominance and luminance components L : lightness (a, b ): chrominance New function to compare luminance values at a given chrominance T q i L H i T q i a, T(q i )[b ] Generalize to a band to consider baseline illumination h i T q i a, T(q i )[b ] T q i L H i T q i a, T(q i )[b ] Incompliant colors Incompliant colors H L a a H Compliant colors Compliant colors Incompliant colors h Note: b omitted for simplicity 33 Compliancy Heightfield Band

9 Presentation Outline Previous work editing overview Formulations of compensation compliancy Solutions to improving/ensuring compliancy Additional tools to improve appearance editing Concluding thoughts and future work 34

10 Augmented AE Pipeline for Compliancy Acquisition & Calibration Light Transport Computation Editing Compensation Edited Scene Scene Projector Placement Modification Target Environment Light 35

11 Projector Placement Planning Goal: discover optimal projector positions to increase available light radiance to improve compliancy Sample each projector s position on imaginary sphere to find optimal combination Photo of Physical Object Photo of Edit Using 3 Projectors (abundant light) Photo of Edit Using Naïve Projector Location Photo of Edit Using Our Optimal Projector Location 36 [Law et al., IEEE Vis 2010]

12 Definition of Optimal Projector Placement Balance of compensation compliancy and light radiance efficiency Light radiance efficiency: low usage of projector illumination power Q k β = β c k + i=1 T(q i )δ i N C max compliancy term β: User specified importance weight c k : compliancy error for visible points + 1 β l k L max N i=1 T(q i )δ i : compliancy error for occluded points l k : average projector intensity value C max, L max : normalizing constants efficiency term 37

13 Placement Algorithm Observation: dense sampling of projector s positions on imaginary sphere is computationally expensive Solution: use iterative adaptive sampling routine θ 38 φ

14 Adaptive Projector Sampling Sparsely sample each projector s position across the specified θ and φ ranges θ 39 φ

15 Adaptive Projector Sampling Sparsely sample each projector s position across the specified θ and φ ranges Can specify ranges in θ and φ where projectors are prohibited θ Observer Viewing 39 φ

16 Adaptive Projector Sampling Select projector combination which yields optimal Q k value θ Observer Viewing 39 φ

17 Adaptive Projector Sampling Select projector combination which yields optimal Q k value Reduce next iteration s sampling range θ Observer Viewing 39 φ

18 Adaptive Projector Sampling Select projector combination which yields optimal Q k value Reduce next iteration s sampling range Sample next iteration (until a specified number of iterations) θ 39 φ

19 Synthetic Example 2 projector setup Target Edit Optimal Ad Hoc Projectors 40

20 Synthetic Example 2 projector setup Edit Compensation Compliancy Light Efficiency Ad Hoc Target not compliant compliant very compliant high light energy low light energy Optimal 40

21 Virtual Restoration Mexican vessels from Casas Grandes Region ( A.D.) Ad Hoc Projectors Optimized Projectors Photo of Original Vessels blurred text, inconsistent colors sharp text, consistent colors Photo of Virtually Restored Vessels blotchy blue color smooth blue color 41 Photo of Visualized Vessels

22 Virtual Restoration Mexican vessels from Casas Grandes Region ( A.D.) Photo of of Edit Edit with with Ad Optimized Hoc Projectors 41

23 California Map Original Map Temperature map visualization with annotations Desired Visualization Achieved Visualization Target Visualization Photo of Ad Hoc Projector Photo of Optimized Projector Ad Hoc Projector Optimized Projector 42

24 Augmented AE Pipeline for Compliancy Acquisition & Calibration Light Transport Computation Editing Compensation Edited Scene Scene Projector Placement Modification Target Environment Light 43

25 Modification Goal: modify colors of target appearance to decrease the amount of light radiance required for compliancy Use CIELAB color space Enables independent control over luminance and chrominance Easy to measure perceptual differences in colors with ΔE Photos of Edits 44 Photo of Physical Object Target Our Perceptually Similar Naïve Incompliant Luminance Reduced [Law et al., CGF 2011, IEEE VR 2012]

26 Perceptually-Based Goals Maintain the chrominance of target colors Maintain the color ratios of target colors Approximation to preserving color constancy Preserving color constancy suggests changes in illuminant, not object Keep loss of luminance to a minimum Bright appearances are desirable 45

27 Scene Deconstruction Target appearance is product of target appearance albedo and shading function Target Target Albedo Shading A patch groups a contiguous set of pixels sharing the same surface albedo color and the same target albedo color a d b c 2 1 (a,1) (a,2) (d,2) (b,1) (c,1) (c,2) Editing Patches 46 Physical Object Surface Patches Target Patches

28 Optimization Select one point r k to represent patch k Point should represent entire patch s compliancy behavior Point classification Compliant point colors Incompliant point colors Barely compliant point colors Incompliant color Barely compliant color L* Compliant color a* τ c Note: b omitted for simplicity 47

29 Optimization Use gradient descent to shift pixels closer to compliancy Patch A Patch B (patch color shifting only) Uses two types of equations: 1. Patch color equations move each individual patch color to a compliant color c k(m+1) = c km + s km 48

30 Optimization Use gradient descent to shift pixels closer to compliancy Constrain with color constancy to preserve perceptual similarity Patch A Patch B (patch color shifting considering ratios) Uses two types of equations: 1. Patch color equations move each individual patch color to a compliant color c k(m+1) = c km + s km 2. Patch ratio equations maintain the color ratios between patches w k1 k 2 s k1 m R k1 k 2 s k2 m = w k1 k 2 R k1 k 2 c k2 m c k1 m 48

31 Optimization Details Discourage change in lightness (L ) Encourage change in the yellow/blue axis (b ) over the red/green axis (a ) Human visual system believed to be less sensitive to shifts in blue/yellow Sun is yellow/white and sky is blue Change in yellow/blue more likely to be perceived as a change in the illuminant instead of a change in the surface 49

32 Perceptual Comparison As compared to reducing luminance (naïve approach) 1. Our method results in brighter scenes with smaller perceptual differences 2. Color constancy is preserved, so perceived differences are even closer than ΔE suggests Our Method Lum. Reduction Scene L* ΔE L* ΔE ped-checker ped-argyle pumpkin house vases Our compute time is Seconds to a few minutes for simple scenes (low number of patches) About one hour for complex scenes (high number of patches) 50

33 Pedestal-Checker Photos of Edits Ideal Albedo Target Perceptually Similar Naïve Incompliant Luminance Reduced Chrominance Zoom-ins Comparison (against ideal) Luminance Comparison (against ideal) (wrong color) (dimmer) 51 Perceptually Similar Naïve Incompliant Perceptually Similar Luminance Reduced

34 Pedestal-Argyle Photos of Edits Physical Object Target Perceptually Similar Naïve Incompliant Luminance Reduced 52 ghosting dimmer

35 House (Extreme Case) (paler) Physical Object Incompliant Target Photo of Naïve Incompliant Edit Zoom-in 53 Ideal Albedo Perceptually Similar Compliant Target Photo of Compliant Edit Zoom-in

36 Editing in Environment Light Advantages Additional light radiance helps achieve spectrally brighter colors Supports scenes with altered and not altered objects Disadvantage Spectrally dark colors now incompliant Use compliancy band to model incompliancy Insufficient projector light Excessive environment light (baseline illumination level too high) 54

37 Modification Augmentation Augment dark room framework with the following Use compliancy band Single photograph E to capture per-pixel environment light New radiometric calibration images captures contributions with both environment light (E) and projector light Use multiple representative points per patch 55

38 Jar with Isolines AE with isolines indicating height up the jar Object Under Environment Light Ideal Target Photo of Ideal Target in Dark Room Photo of Ideal Target in Lit Room Our Target Photo of Our Target In Lit Room 56

39 United States Map Visualization of areas at risk due to rising sea levels Ideal Target Photo of Ideal Target in Dark Room Map Under Environment Light Photograph of our Target in Lit Room 57 Our Target Photo of Our Target in Lit Room

40 Car Aerodynamics Car model augmented with aerodynamic lines Car Under Environment Light Zoom-in Ideal Target Photo of Ideal Target in Dark Room Error Zoom-in 58 Our Target Photo of Our Target in Lit Room Error

41 <slides removed>