Numerical Derivatives

Transcription

1 Lab 15 Numerical Derivatives Lab Objective: Understand and implement finite difference approximations of te derivative in single and multiple dimensions. Evaluate te accuracy of tese approximations. Ten use finite difference quotients to find edges in images via te Sobel filter. Derivative Approximations in One Dimension Te derivative of a function f at a point x 0 is f f(x 0 + ) f(x 0) (x 0) = lim. (15.1) 0 In tis lab, we will investigate one way a computer can calculate f (x 0). Forward Difference Quotient Suppose tat in Equation (15.1), instead of taking a limit, we just pick a small value for. Ten we would expect f (x 0) to be close to te quantity f(x 0 + ) f(x 0). (15.) Tis quotient is called te first order forward difference approximation of te derivative. Because f (x 0) is te limit of suc quotients, we expect tat wen is small, tis quotient is close to f (x 0). We can use Taylor s formula to find just ow close. By Taylor s formula, f(x 0 + ) = f(x 0) + f (x 0) + R (), ( ) 1 were R () = (1 t)f (x t)dt. (Tis is called te integral form of te remainder for Taylor s Teorem; see Volume 1 Capter 6). Wen we solve tis equation for f (x 0), we get f f(x0 + ) f(x0) (x 0) = R(). (15.3) Tus, te error in using te first order forward difference quotient to approximate f (x 0) is R () 1 1 t f (x 0 + t) dt

2 164 Lab 15. Numerical Derivatives If we assume f is continuous, ten for any δ, set M = sup x (x0 δ,x 0 +δ) f (x). Ten if < δ, we ave R() 1 Mdt = M O(). Terefore, te error in using (15.) to approximate f (x 0) grows like. Centered Difference Quotient 0 In fact, we can approximate f (x 0) to te second order wit anoter difference quotient, called te centered difference quotient. We begin by trying to find te backward difference quotient. Evaluate Taylor s formula at x 0 to derive f (x 0) = f(x0) f(x0 ) + R( ). (15.4) Te first term on te rigt and side of (15.4) is called te backward difference quotient. Tis quotient also approximates f (x 0) to first order, so it is not te quotient we are looking for. Wen we add (15.3) and (15.4) and solve for f (x 0) (by dividing by ), we get f 1 (x 0) = f(x0 + ) 1 f(x0 ) R( ) R() + (15.5) Te centered difference quotient is te first term of te rigt and side of (15.5). Let us investigate te remainder term to see ow accurate tis approximation is. Recall from te proof of Taylor s teorem tat R k = f (k) (x 0 ) R ( ) R () = 1 ( f (x 0) k + R k! k+1. Terefore, + R 3( ) f ) (x 0) R 3() = 1 (R3( ) R3()) = 1 (( 1 (1 t) f (x 0 + t)dt 0 ( 1 (1 t) = 4 0 O( ) ) ( 1 3 ) (f (x 0 + t) f (x 0 t)) 0 (1 t) ) ) f (x 0 t)dt 3 once we restrict to some δ-neigborood of 0. So te error in using te centered difference quotient to approximate f (x 0) grows like, wic is smaller tan wen < 1. Accuracy of Approximations Let us discuss wat step size we sould plug into te difference quotients to get te best approximation to f (x 0). Since f is defined as a limit as 0, you may tink tat it is best to coose as small as possible, but tis is not te case. In fact, dividing by very small numbers causes errors in floating point aritmetic. Tis means tat as we decrease, te error between f (x 0) and te difference quotient will first decrease, but ten increase wen gets too small because of floating point aritmetic. Here is an example wit te function f(x) = e x. A quick way to write f as a function in Pyton is wit te lambda keyword. >>> import numpy as np >>> from matplotlib import pyplot as plt >>> f = lambda x: np.exp(x)

3 165 1e-1 1e-3 1e-5 1e-7 1e-9 1e-11 Error 5e-3 5e-7 6e-11 6e-11 7e-9 1e-5 Table 15.1: Tis table sows tat it is best not to coose too small wen you approximate derivatives wit difference quotients. Here, Error equals te absolute value of f (1) f app (1) were f(x) = e x and f app is te centered difference approximation to f. In general, te line f = lambda <params> : <expression> is equivalent to defining a function f tat accepts te parameters params and returns expression. Next we fix a step size and define an approximation to te derivative of f using te centered difference quotient. >>> = 1e-1 >>> Df_app = lambda x:.5*(f(x+)-f(x-))/ Finally, we ceck te accuracy of tis approximation at x 0 = 1 by computing te difference between Df_app(1) and te actual derivative evaluated at 1. # Since f(x) = e^x, te derivative of f(x) is f(x) >>> np.abs( f(1)-df_app(1) ) We note tat our functions f and Df_app beave as expected wen tey are passed a NumPy array. >>> = np.array([1e-1, 1e-3, 1e-5, 1e-7, 1e-9, 1e-11]) >>> np.abs( f(1)-df_app(1) ) array([ e-03, e-07, e-11, e-11, e-09, e-05]) Tese results are summarized in Table Tus, te optimal value of is one tat is small, but not too small. A good coice is = 1e-5. Problem 1. Write a function tat accepts as input a callable function object f, an array of points pts, and a keyword argument tat defaults to 1e-5. Return an array of te centered difference quotients of f at eac point in pts wit te specified value of. You may wonder if te forward or backward difference quotients are ever used, since te centered difference quotient is a more accurate approximation of te derivative. In fact, tere are some functions tat in practice do not beave well under centered difference quotients. In tese cases, one must use te forward or backward difference quotient. Finally, we remark tat forward, backward, and centered difference quotients can be used to approximate iger-order derivatives of f. However, taking derivatives is an unstable operation. Tis means tat taking a derivative can amplify te aritmetic error in your computation. For tis reason, difference quotients are not generally used to approximate derivatives iger tan second order.

4 166 Lab 15. Numerical Derivatives Derivative Approximations in Multiple Dimensions Finite difference metods can also be used to calculate derivatives in iger dimensions. Recall tat te Jacobian of a function f : R n R m at a point x 0 R n is te m n matrix J = (J ij) defined component-wise by J ij = i x j (x 0). For example, te Jacobian for a function f : R 3 R is defined by J = ( x 1 x x 3 ) = ( 1 x 1 1 x 1 x 3 x 1 x x 3 Te Jacobian is useful in many applications. For example, te Jacobian can be used to find zeros of functions in multiple variables. Te forward difference quotient for approximating a partial derivative is x j (x 0) f(x0 + ej) f(x0), were e j is te j t standard basis vector. Similarly, te centered difference approximation is 1 (x 0) f(x0 + ej) 1 f(x0 ej). x j ). Problem. Write a function tat accepts 1. a function andle f,. an integer n tat is te dimension of te domain of f, 3. an integer m tat is te dimension of te range of f, 4. an 1 x n-dimensional NumPy array pt representing a point in R n, and 5. a keyword argument tat defaults to 1e-5. Return te approximate Jacobian matrix of f at pt using te centered difference quotient. Problem 3. Let f : R R be defined by ( ) e x sin(y) + y 3 f(x, y) = 3y cos(x) Find te error between your Jacobian function and te analytically computed derivative on te square [ 1, 1] [ 1, 1] using ten tousand grid points (100 per side). You may apply your Jacobian function to te points one at a time using a double for loop. Once you get te error matrix for a given point, calculate te Frobenius norm of tis matrix (la.norm defaults to te Frobenius norm). Tis norm will be your total error for tat point. Wat is te maximum error of your Jacobian function over all points in te square?

5 167 ( ) x Hint: Te following code defines te function f(x, y) =. x + y # f accepts a lengt- NumPy array >>> f = lambda x: np.array([x[0]**, x[0]+x[1]]) Application to Image Filters Recall tat a computer stores an image as a -D array of pixel values (i.e., a matrix of intensities). An image filter is a function tat transforms an image by operating on it locally. Tat is, to compute te ij t pixel value in te new image, an image filter uses only te pixels in a small neigborood around te ij t pixel in te original image. In tis lab, we will use a filter derived from te gradient of an image to find edges in an image. Convolutions One example of an image filter is to convolve an image wit a filter matrix. A filter matrix is a matrix wose eigt and widt are relatively small odd numbers. If te filter matrix is f 1, 1 f 1,0 f 1,1 F = f 0, 1 f 0,0 f 0,1, f 1, 1 f 1,0 f 1,1 ten te convolution of an image A wit F is A F = (C ij) were C ij = 1 k= 1 l= 1 1 f kl A i+k,j+l. (15.6) Say A is an m n matrix. Here, we take A ij = 0 wen i {1,... m} or j {1,..., n}. Te value of C ij is a linear combination of te nearby pixel values, wit coefficients given by F (see Figure 15.1). In fact, C ij equals te Frobenius inner product of F wit te 3 3 submatrix of A centered at ij. Implementation in NumPy Let us write a function tat convolves an image wit a filter. You can test tis function on te image cameraman.jpg, wic appears in Figure 15.a. Te following code loads tis image and plots it wit matplotlib. >>> image = plt.imread('cameraman.jpg') >>> plt.imsow(image, cmap = 'gray') >>> plt.sow() Here is te function definition and some setup. 1. def Filter(image, F):. m, n = image.sape 3., k = F.sape

6 168 Lab 15. Numerical Derivatives Figure 15.1: Tis diagram illustrates ow to convolve an image wit a filter. Te ligt grey rectangle represents te original image A, and te dark grey squares are te filter F. Te larger rectangle is te image padded wit zeros; i.e., all pixel values in te outer wite band are 0. To compute te entry of te convolution matrix C located at a black dot, take te inner product of F wit te submatrix of te padded image centered at te dot. To convolve image wit te filter F, we must first pad te array image wit zeros around te edges. Tis is because in (15.6), entries A ij are set to zero wen i or j is out of bounds. We do tis by creating a larger array of zeros, and ten making te interior part of te array equal to te original image (see Figure 15.1). For example, if te filter is a 3 3 matrix, ten te following code will pad te matrix wit te appropriate number of zeros. # Create a larger matrix of zeros image_pad = np.zeros((m+, n+)) # Make te interior of image_pad equal to te original image image_pad[1:1+m, 1:1+n] = image We want to do tis in general in our function. Note tat te number of zeros we need to pad our array depends on te size of te filter F. 5. image_pad = # Create an array of zeros of te appropriate size 6. # Make te interior of image_pad equal to image Finally, we iterate troug te image to compute eac entry of te convolution matrix. 7. C = np.zeros(image.sape) 8. for i in range(m): 9. for j in range(n): 10. C[i,j] = # Compute C[i, j]

7 169 Gaussian Blur A Gaussian blur is an image filter tat operates on an image by convolving wit te matrix G = Blurring an image can remove noise, or random variation tat is te visual analog of static in a radio signal (and equally undesirable). Problem 4. Finis writing te function Filter by filling in lines 5, 6, and 10. Hint: Note in 15.6, C ij was calculated by summing from -1 to 1. Tis is only te case if te filter F is 3 3. A sligt modification is needed in te general case. Test your function on te image cameraman.jpg using te Gaussian Blur. Te result is in Figure 15.b. Edge Detection Automatic detection of edges in an image can be used to segment or sarpen te image. We will find edges wit te Sobel filter, wic computes te gradient of te image at eac pixel. Te magnitude of te gradient tells us te rate of cange of te pixel values, and so large magnitudes sould correspond to edges witin te image. Te Sobel filter is not a convolution, altoug it does use convolutions. We can tink of an image as a function from a grid of points to R. Te image maps a pixel location to an intensity. It does not make sense to define te derivative of tis function as a limit because te domain is discrete a step size cannot take on arbitrarily small values. Instead, we define te derivative to be te centered difference quotient of te previous section. Tat is, we define te derivative in te x-direction at te ij t pixel to be 1 Ai+1,j 1 Ai 1,j. We can use a convolution to create a matrix A x wose ij t entry is te derivative of A at te ij t entry, in te x-direction. In fact, A x = A S, were S = Note tat tis convolution takes a weigted average of te x-derivatives at (i, j), (i, j + 1), and (i, j 1). Te derivative at (i, j) is weigted by. Using a weigted average instead of just te derivative at (i, j) makes te derivative less affected by noise. Now we can define te Sobel filter. A Sobel filter applied to an image A results in an array B = (B ij) of 0 s and 1 s, were te 1 s trace out te edges in te image. By definition, { 1 if A(ij) > M B ij = 0 oterwise. Here, A(ij) = ((A S) ij, (A S T ) ij) is te gradient of A at te ij t pixel. Te constant M sould be sufficiently large enoug to pick out tose pixels wit te largest gradient

8 170 Lab 15. Numerical Derivatives (a) Unfiltered image. (b) Image after Gaussian blur is applied. (c) Image after te Sobel filter is applied. Figure 15.: Here is an example of a Gaussian blur and te Sobel filter applied to an image. Tis poto, known as cameraman, is a standard test image in image processing. A database of suc images can be downloaded from ttp:// (i.e., tose pixels tat are part of an edge). A good coice for M is 4 times te average value of A(ij) over te wole image A. Wen te Sobel filter is applied to cameraman.jpg, we get te image in Figure 15.c. Here, te 1 s in B were mapped to wite and te 0 s were mapped to black. Problem 5. Write a function tat accepts an image as input and applies te Sobel filter to te image. Test your function on cameraman.jpg. Hint: If you want to find te average of a matrix A, use te function A.mean().