Programming and Pictures: Computational Photography and Applications to Math and Physics. GISA 2016 Session 131 Marist School Christopher Michaud

Transcription

1 Programming and Pictures: Computational Photography and Applications to Math and Physics GISA 2016 Session 131 Marist School Christopher Michaud

2 Purpose Explore how math and physics concepts are used to create visual effects in Image Editing

3 Why? Our society is a creature with billions of eyes. Digital image editing and sharing play a significant role in social media systems. Our students document and live through their pictures. We want to connect the world of digital images to the concepts in math and science to increase fluency and build connections beyond the symbols on the page.

4 Goals We want our students to visualize the role of Rate of Change and how the derivative forms the basis of human perception in images and sound. Change how our students see and experience the world through a visual and kinetic understanding of functions. Bring models and equations to life through image manipulation.

5 Elements of Computational Photography 1. Illumination 2. Optics 3. Sensor 4. Processing 5. Display 6. User Source: Irfan Essa, Georgia Tech 2015

6 Pictures as Functions: Python

7 Modeling Images as Functions: One Channel I =

8 Modeling Images as Functions (Color: BGR) I =

9 Pictures as Functions and numpy arrays: Python Color Images Mathematical Expression: I(100, 50) = {61, 150, 184} Python Expression img[50][100] # Returns: array([ 61, 150, 184], dtype=uint8) NOTE! In Mathematical expressions we write I(x, y) In programming (most languages) we write image arrays as: img_array[row][column] -That becomes img_array[y][x]

10 Pictures as Functions and numpy arrays: Grayscale images Mathematical Expression: I(100, 50) = 150 Python Expression img_g[50][100] # Returns value of 150 NOTE! In Mathematical expressions we write I(x, y) In programming (most languages) we write image arrays as: img_array[row][column] -That becomes img_array[y][x]

11 Numpy Arrays in Python Numpy Arrays can be written two ways: Nested: img[50][100][3] Means Row Index 50, Column Index 100, Blue Channel Vectorised img[50, 100, 3]

12 Why Vectors? Can call range of Values without For Loops Slow Way: Iteration # Nested For Loop part_img = [[]] fill = [] for r in range (50, 100): fill = [] for c in range(60, 80): fill.append(img_g[r][c]) part_img.append(fill) Fast Way: Vectors and Numpy part_img = img_g[50:100, 60:80] # Yes this one line is equal to # the nested for loop on the left! # And, this is much faster to # to process needed for real time # Movie processing

13 Some Python Examples:

14 Function Calls in Example Code Function normalize(img) adjustcolors(img, red, green, blue) avgcolors(img) getdx(img) getdy(img) getgradient(img) avgtwopics(img1, img2) getlines(img) Description Fits data between 0.0 and 1.0. Returns image Adjusts values in color channel by scalar multiplication. Returns image Averagres values in Red, Green, Blue color channels and returns greyscale image. Returns the derivative (Rate of Change) of pixel intensity in x direction. Returns the derivative (Rate of Change) of pixel intensity in y direction. Returns magnitude and angles of image gradient. Returns the average of two images. (Mean at each pixel) Returns the lines found in image.

15 Vectorised Array: Set Green and Blue to 0

16 Vectorised Array: Divide entire picture by 2

17 Definition of a Derivative Rate of Change over between 2 or move variables. Expressed as a ratio yy xx Slope of a Line tangential to a point on a continuous function Foundation of Calculus / Physics / AI Continuous and Discrete applications.

18 You have already used Derivatives with Linear Functions. The Derivative in a linear function is the same as the slope.

19 Mathematical Definition of a Derivative (Continuous Functions) The derivative of a f(x) with respect to x is the function f (x) and is defined as: ff xx = lim h 0 ff xx + h ff(xx) h

20 For most continuous functions, we can compute the derivatives with formulas.

21 Another Continuous Function

22 What about discrete data? In computing (such as images or sound), Data is discrete. Comes in Chunks May not be continuous or defined by formula We can still compute and use derivatives. How? Element wise subtraction!

23 Example Data: Row 328 What does the rate of change tell us about this data? [216, 125, 86, 109, 156, 156, 76, 50, 48, 34, 61, 134, 97]

24 Example: Row 328 [:] What could the derivative of this data tell us? What information can we gather?

25 What can we get from the derivative?

26 What can we get from the derivative?

27 Closer Look...

28 Derivatives for Two Inputs: Image Functions Recall the Intensity Function: II(xx, yy) Returns the intensity of a pixel value at (x,y) We can take two derivatives For change in intensity / change in x For change in intensity / change in y

29 Continuous Function I(x, y) Two Derivatives: Formal Definition for x direction: II(xx, yy) xx = lim h 0 ff xx + h, yy ff(xx, yy) h Formal Definition for y direction: II(xx, yy) yy = lim h 0 ff xx, yy + h ff(xx, yy) h

30 But.. Images are discrete. Thus, we modify for discrete data: X Direction II(xx, yy) xx = ff xx + 1, yy ff(xx, yy) 1 II(xx,yy) xx = ff xx + 1, yy ff(xx, yy)

31 But.. Images are discrete. Thus, we modify for discrete data: Y Direction II(xx, yy) yy = ff xx + 1, yy ff(xx, yy) 1 II(xx,yy) yy = ff xx, yy + 1 ff(xx, yy)

32 What does this look like in an actual image? II(xx,yy) xx = ff xx + 1, yy ff(xx, yy)

33 What does this look like in an actual image? II(xx,yy) yy = ff xx, yy + 1 ff(xx, yy)

34 Side by Side comparison... II(xx, yy) xx II(xx, yy) yy

35 Derivate in x with discrete data (One Dimension): VV = VV =

36 Derivate in x with discrete data (One Dimension): VV = VV = ?

37 Application: Image Gradients = II xx, II yy θθ = II xx, 0 II = 0, yy = II xx, II yy Gradient points in direction of the most rapid change in Intensity.

38 Mathematical Definitions: The gradient of an Image: = II xx, II yy The gradient Direction: θθ = tan 1 II yy / II xx The gradient Magnitude (Edge Strength): = II xx 2 + II yy 2

39 Python Example: Derivative on one dimension

40 Correlation and Convolution Filters Used to Filter Images Blurring Effects Remove Noise Prepare Images for further analysis

41

42

43 2D Filter Function (Convolution) cv2.filter2d(img, -1, kernel) Takes input of grayscale img data filters with kernel. The kernel is an m x m array used to filter the data. Usually a Gaussian, Laplacian, Sorbel, or box filter. Example: # img is an existing numpy image array kernel = np.ones((5, 5), np.float32)/25 box_blur = cv2.filter2d(img, -1, kernel)

44 Gaussian Blur: Role of Sigma We will blur images with the cv2.gaussianblur() function. cv2.gaussianblur(img, (m x m), sigma) Where: img = Image Array to Blur (m x m): size of window of Gaussian (odd number) sigma: The spread of the blur (High for more depth)

45 CV2 Canny Edge Function Built in Edge detection function in Open CV that uses a more involved algorithm: Filters image with derivative of Gaussian Uses magnitude and orientation of gradients Non-Maximum suppression Linking and thresholding Result is a more refined edge image. img_edge_canny = cv2.canny(img_gauss, 50, 100)

46 2D Gaussian Filter Image

47 Example:

48 Why Blur with Gaussian? Remove Noise from the Picture.

49 Code Sample: Blur with Gaussian

50 Code Sample: Canny Edge Note: You will have to experiment with the parameters for GaussianBlur() and Canny() to get a good edge image.

51 Template Matching with Python and Open CV Goal: With a patch or section of image, search a larger image or set of data and find the closest match. Foundation of recognition. Two types to consider: Normalized Correlation Sum of Squared Differences

52 Sum of Squared Differences Note: This formula is based on center matching where the image x is searched by template t around the center point of the search.

53 SSD Example: (Corner Matching) img patch

54 SSD Example img patch

55 SSD Example Non Normalized

56 SSD Example Normalized / 63

57 Note that the filters find the Upper Left Corner of the Match

58 What about an in-exact match? Still Works! Closet Match found

59 Example with an Image patch Source: lisa.png img

60 Correlation Map Brightest area is region of best match. patch img f (Correlation Map)

61 Difference Map Darkest area is region of best match. patch img f (Correlation Map)

62 Not Exact Match: Last Supper Note that it did find a face. Correlations and SSD are not perfect, but they provide a starting point for Computer vision

63 Normalized Correlation

64 Algorithm: Normalized Correlation 1. Convert image and patch to grayscale 2. Filter image with patch -> f 3. Locate row and column in f with highest value -> point 4. Identify x and y from point object 5. Identify width and height from patch 6. Draw Rectangle at x, y with width and height

65 Python, CV2, and Numpy commands for Template matching f = cv2.matchtemplate(image, patch, MatchType) matchtemplate() returns a correlation map using the MatchType constant. -patch refers to the search template -image: Image array to be searched or filtered (Examples of MatchType) TM_CCORR_NORMED TM_SQDIFF_NORMED TM_SQDIFF Example: f = cv2.matchtemplate(img, patch, cv2. TM_CCORR_NORMED)

66 Python, CV2, and Numpy commands for Template matching np.where(f == value) np.where() returns a list of points from an array based on the search parameters point = np.where(f == f.max()) y = point[0][0] x = point[1][0]

67 Code Samples: Sum of Squared Differences # img is the image to search # patch is the template # Make Grayscale gray = cv2.cvtcolor(img, cv2.color_bgr2gray) # Create Summed of Squared Differences with patch ssd = cv2.matchtemplate(gray, patch, cv2.tm_sqdiff_normed) # find min for match with SQDIFF_NORMED point = np.where(ssd == ssd.min()) y = point[0][0] x = point[1][0] w = len(patch[0]) l = len(patch) # Draw Rectangle cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2)

68 Code Samples: Normalized Correlation # img is the image to search # patch is the template # Make Grayscale gray = cv2.cvtcolor(img, cv2.color_bgr2gray) # Create Normalized Correlation with Patch f = cv2.matchtemplate(gray, patch, cv2.tm_ccorr_normed) # Find max for match with CCORR_NORMED point = np.where(f == f.max()) y = point[0][0] x = point[1][0] w = len(patch[0]) h = len(patch) #Draw Rectangle cv2.rectangle(img, (x, y), (x+w, y+h), (0, 0, 255), 2)