2%34 #5 +,,% ! # %& ()% #% +,,%. & /%0%)( 1 ! # %&# (&)# +, %& ./01 /1&2% # /&

Transcription

1 ! # %& ()% #% +,,%. & /%0%)( 1 2%34 #5 +,,%! # %&# (&)# +, %&./01 /1&2% # /& & Many slides in this lecture are due to other authors; they are credited on the bottom right

2 &. 4. # 5 9% #:5 # #%,.7 ;

3 34%&( %# &15&#& #)4/ %&+4#+&)# &6%%&

4 2/ 6+&% )1, +% & 74#+&(1&81,&6%%&9&

5 2/ 6+&% )1, +% & : (& 1;&9&

6 2/ 6+&% )1, +% & <# /(&=/6,#>& %)1 /?1 &/6&/ 1 +# +& &

7 34%& 1#>&15&)1,+% &=/6/1 & 31&%Α+ #)+&Β %# / Χ&5 1 & /Α%>6& Humans are remarkably good at this Source: 80 million tiny images by Torralba et al.

8 74#+& / (&15&/ 51 #?1 &)# &Ε%& %Α+ #)+%(&5 1 &# &/ # %9& ΦΓ& %#6, % % +6& 6% #?)6&

9 ./6/1 &#6& %#6, % % +&(%=/)%& Real-time stereo Structure from motion Reconstruction from Internet photo collections NASA Mars Rover Pollefeys et al. Goesele et al.

10 Vision as a source of semantic information slide credit: Fei-Fei, Fergus & Torralba

11 Object categorization sky building flag banner bus face street lamp bus wall cars slide credit: Fei-Fei, Fergus & Torralba

12 Scene and context categorization outdoor city traffic slide credit: Fei-Fei, Fergus & Torralba

13 Qualitative spatial information slanted non-rigid moving object vertical rigid moving object horizontal rigid moving object slide credit: Fei-Fei, Fergus & Torralba

14 748&/6&)1,+% &=/6/1 &(/Η),>+9&

15 Challenges: viewpoint variation Michelangelo slide credit: Fei-Fei, Fergus & Torralba

16 Challenges: illumination image credit: J. Koenderink

17 Challenges: scale slide credit: Fei-Fei, Fergus & Torralba

18 Challenges: non-rigid deformation Xu, Beihong 1943 slide credit: Fei-Fei, Fergus & Torralba

19 Challenges: occlusion Magritte, 1957 slide credit: Fei-Fei, Fergus & Torralba

20 Challenges: background clutter

21 Challenges: object intra-class variation slide credit: Fei-Fei, Fergus & Torralba

22 ! 4% % +&# Ε/,/+8&15&+4%& 1Ε>% & Ι# 8&(/ϑ% % +&ΦΓ&6)% %6&)1,>(&4#=%& /=% & /6%&+1&#& #?),># & ΚΓ&/ # %& & & & & & & & Λ166/Ε>%&61>,?1 6& Μ / &/ & 1 %&)1 6+ #/ +6&Ν 1 %&/ # %6Ο& Π6%& /1 & 1;>%( %&#Ε1,+&+4%&6+,)+, %&15&+4%&;1 >(& Θ%%(&#&)1 Ε/ #?1 &15& %1 %+ /)&# (&6+#?6?)#>& %+41(6& slide credit: Lazebnik

23 Ρ1 %)?1 6&+1&1+4% &(/6)/ >/ %6& Artificial Intelligence Robotics Machine Learning Computer Vision Computer Graphics Cognitive science Neuroscience Image Processing slide credit: Lazebnik

24 A first concrete image processing program in Matlab: Distribution of pixel values Slide credit: Bob Fisher

25 Matlab for image read and display Can also use emacs on files in another window. Slide credit: Bob Fisher

26 Figure output Use File >Export to save *.eps files for printing and documents Slide credit: Bob Fisher

27 Matlab in command window bigf = myjpgload( partbigf,3); [H,W] = size(bigf) H = 384 W = 510 figure(3) % what the 3 above does colormap(gray) % " imagesc(bigf) % " Slide credit: Bob Fisher

28 thehist = zeros(256,1); [H,W] = size(bigf); bigf histogram for r = 1 : H for c = 1 : W value = round(bigf(r,c)); end if value < 0 % array goes 1:256 value = 0; % but image goes 0:255 elseif value > 255 value = 255; end thehist(value+1) = thehist(value+1) + 1; end Slide credit: Bob Fisher

29 figure(4) plot(thehist) axis([0, 255, 0, 1.1*max(thehist)]) Slide credit: Bob Fisher

30 histc histogram builtin % set up bin edges for histogram edges = zeros(256,1); for i = 1 : 256 edges(i) = i-1; end [R,C] = size(bigf); imagevec = reshape(bigf,1,r*c); % make long array thehist = histc(imagevec,edges) ; % do histog. figure(1) plot(thehist) axis([0, 255, 0, 1.1*max(thehist)]) Slide credit: Bob Fisher

31 Histogram Output x Slide credit: Bob Fisher

32 Midlecture Problem Why does the histogram have this shape? x Slide credit: Bob Fisher

33 xv entry screen Slide credit: Bob Fisher

34 xv control panel Slide credit: Bob Fisher

35 Basic xv actions Make control panel appear/disappear: right click on display Make image editor appear/disappear: type e in display Load/Save image: left click Load/Save button Grab from screen: left click Grab button Crop: left click and drag on display, left click on crop Shrink/Expand 10%: type,/. Shrink 1/2 / Double: type < / > Slide credit: Bob Fisher

36 Flat Part Recognition How to recognize these and similar parts Assumptions: Flat, viewed orthographically Good contrast everywhere No specularities Slide credit: Bob Fisher

37 Flat part recognition algorithm 1. Capture image 2. Extract object 3. Compute properties 4. Use properties to compute class 5. Learning model properties for the classes Lecture Plan: Covering each of the steps in more detail with theory, Matlab code and examples Slide credit: Bob Fisher

38 Top level Matlab Dim = 3; % number of feature properties modelfile = input( Model file name\n?, s ); eval([ load,modelfile, NumCls Means ICor ]) run=1; while ~(run == 0) currentimagergb = liveimagejpg currentimage = rgb2gray(currentimagergb); vec = extractprops(currentimage); class = classify(vec,numcls,means,icor,dim) run = input( Do another image (0,1)\n? ); end Slide credit: Bob Fisher

39 Image capture: elementary physics LIGHT SOURCE SCENE SENSOR Figure by Bob Fisher

40 Let s design a camera Idea 1: put a piece of film in front of an object Do we get a reasonable image? Slide by Steve Seitz

41 Pinhole camera Add a barrier to block off most of the rays This reduces blurring The opening is known as the aperture Slide by Steve Seitz

42 Pinhole camera model Pinhole model: Captures pencil of rays all rays through a single point The point is called Center of Projection (focal point) The image is formed on the Image Plane Slide by Steve Seitz

43 Dimensionality reduction: from 3D to 2D 3D world 2D image Point of observation What is preserved? Straight lines, incidence What have we lost? Angles, lengths Slide by A. Efros Figures Stephen E. Palmer, 2002

44 Projection properties Many-to-one: any points along same visual ray map to same point in image Points points But projection of points on focal plane is undefined Lines lines (collinearity is preserved) But lines through focal point (visual rays) project to a point Planes planes (or half-planes) But planes through focal point project to lines slide credit: Lazebnik

45 Vanishing points Each direction in space has its own vanishing point All lines going in that direction converge at that point Exception: directions parallel to the image plane slide credit: Lazebnik

46 Modeling projection f y z x The coordinate system The optical center (O) is at the origin The image plane is parallel to xy-plane (perpendicular to z axis) Source: J. Ponce, S. Seitz

47 Modeling projection f y z Projection equations Compute intersection with image plane of ray from P = (x,y,z) to O Derived using similar triangles x x y ( x, y, z) ( f, f, f z z We get the projection by throwing out the last coordinate: ( x, y, z) ( f x z, f y ) z ) Source: J. Ponce, S. Seitz

48 Homogeneous coordinates x y ( x, y, z) ( f, f ) z z Is this a linear transformation? no division by z is nonlinear Trick: add one more coordinate: homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates Slide by Steve Seitz

49 Perspective Projection Matrix Projection is a matrix multiplication using homogeneous coordinates

50 divide by the third coordinate Perspective Projection Matrix Projection is a matrix multiplication using homogeneous coordinates = f z y x z y x f / 1 0 1/ ), ( z y f z x f

51 Perspective Projection Matrix Projection is a matrix multiplication using homogeneous coordinates x 0 0 x y 0 0 = y z 1/ f 0 z/ f 1 In practice: lots of coordinate transformations ( f x z, f y z ) divide by the third coordinate 2D point (3x1) = Camera to pixel coord. trans. matrix (3x3) Perspective projection matrix (3x4) World to camera coord. trans. matrix (4x4) 3D point (4x1)

52 Image Capture: Camera basics PHOTOSENSITIVE IMAGING SURFACE FOCUSSABLE LENS DIGITIZER PC Cameras: webcam (c pounds). Machine vision ( pounds). Digitizer: comes with webcam/interface. It handles interlace, video conventions. Various PC peripheral interfaces. Only consider details for serious vision work. Slide credit: Bob Fisher

53 Image Capture: Photon capture TYPICAL SOLID STATE SENSOR RELATIVE SENSITIVITY VISIBLE INFRARED B R WAVELENGTH 1100 Sensitivity varies by wavelength Sensitivity beyond (human) visible range Slide credit: Bob Fisher

54 Image Capture: Photon readout PIXEL PIXEL TIME 1a 1b + 1c 2a 2b + 2c t1 e e + + t2 e e t3 e e e... Photons converted to electrons Shift electrons along row for readout Three sets for 3 colours: red/green/blue Slide credit: Bob Fisher

55 Image Capture: Matlab % capture a 640x480 jpg color image and return it function Im = liveimagejpg(filename) unix( mplayer tv:// -tv... driver=v4l2:width=640:height=480:... device=/dev/video0... -frames 5 -vo jpeg ); unix([ mv jpg, filename,.jpg ]) Im=imread(filename,.jpg ], jpg ); See: man mplayer Slide credit: Bob Fisher

56 Color spaces: RGB space 3 primaries are monochromatic lights (for monitors, correspond to three types of phosphors) Linearly combined to produce other colors Unnatural to manipulate for humans, but good for computers to produce color RGB primaries

57 Color spaces: HSV space (nonlinear) Perceptually meaningful dimensions: Hue, Saturation, Value (Intensity) RGB cube on its vertex

58 Image Capture and Problems A reasonable capture Slide credit: Bob Fisher

59 Image Capture: Focus problems Focus set to one distance. Nearby distances in focus (depth of focus). Further not so well focused. Compare identical lines. Slide credit: Bob Fisher

60 Image Capture: Shadow problems False color to emphasize the shadow location. Often hard to separate from part. Slide credit: Bob Fisher

61 Image Capture: Saturation problems Pixels clip at 255. Slide credit: Bob Fisher

62 Image Capture: Specularities/highlights Saturated pixels set to red. Slide credit: Bob Fisher

63 Image Capture: Non-uniform illumination Contrast on background enhanced: may cause analysis problems. Slide credit: Bob Fisher

64 Image Capture: Radial lens distortion Note straight lines at image edge. May make accurate measurements hard. Slide credit: Bob Fisher

65 Image Capture: Overcoming Problems Shadows, specularities, non-uniform illumination: increase ambient lighting by using light diffusing panels or lots of point lights Depth of Focus: use smaller aperture and brighter light Motion Blur: use shorter capture time and brighter light Saturation: use smaller aperture, reduce gain and adjust gamma Slide credit: Bob Fisher

66 Lens Distortion: more expensive lenses, view from further away Aliasing: use incandescent lights Slide credit: Bob Fisher

67 Illumination control techniques Main cause of problem: point light sources Brightness = B / (surface distance from source) 2 Sharp shadows: Strong illumination variations Slide credit: Bob Fisher

68 Shadow Example Figure and shadow at bottom left emphasized Slide credit: Bob Fisher

69 Lighting control To reduce complications arising from illumination: Increase ambient (all direction) light with light diffuser panels Illumination by camera to move shadows to non-visible places Backlighting panel Slide credit: Bob Fisher

70 LIGHTS NEAR CAMERA DIFFUSER PANEL MUCH LESS SHADOW Slide credit: Bob Fisher