Optimization Methods: Optimization using Calculus-Stationary Points 1 Module - Lecture Notes 1 Stationary points: Functions of Single and Two Variables Introduction In this session, stationary points of a function are defined. The necessary and sufficient conditions for the relative maimum of a function of single or two variables are also discussed. The global optimum is also defined in comparison to the relative or local optimum. Stationary points For a continuous and differentiable function f() a stationary point * is a point at which the slope of the function vanishes, i.e. f () = at = *, where * belongs to its domain of definition. minimum maimum Fig. 1 inflection point A stationary point may be a minimum, maimum or an inflection point (Fig. 1). Relative and Global Optimum A function is said to have a relative or local minimum at = * if * * f ( ) f( + h) for all sufficiently small positive and negative values of h, i.e. in the near vicinity of the point *. Similarly a point * is called a relative or local maimum if f f h * * ( ) ( + ) for all values of h sufficiently close to zero. A function is said to have a global or absolute minimum at = * if f * ( ) f( ) for all in the domain over which f() is defined. Similarly, a function is ML1
Optimization Methods: Optimization using Calculus-Stationary Points said to have a global or absolute maimum at = * if f * ( ) f( ) for all in the domain over which f() is defined. Figure shows the global and local optimum points. A 1, A, A 3 = Relative maima A = Global maimum B 1, B = Relative minima B 1 = Global. minimum A f() f(). Relative minimum is also.. global optimum (since only one minimum point is there) A 3 A 1 B.. B 1 a b a b Fig. Functions of a single variable Consider the function f() defined for a b. To find the value of * [ ab, ] such that = * maimizes f() we need to solve a single-variable optimization problem. We have the following theorems to understand the necessary and sufficient conditions for the relative maimum of a function of a single variable. Necessary condition: For a single variable function f() defined for [ ab, ] which has a relative maimum at = *, * [ ab, ] if the derivative f '( X) = df( ) / deists as a finite number at = * then f ( * ) =. This can be understood from the following. ML1
Optimization Methods: Optimization using Calculus-Stationary Points 3 Proof. Since f ( * ) is stated to eist, we have ' f ( * + h) f( *) f (*) = lim (1) h h From our earlier discussion on relative maima we have f ( *) f ( * + h) for h. Hence f( * + h) f( *) h < () h f( * + h) f( *) h > (3) h which implies for substantially small negative values of h we have f ( *) and for substantially small positive values of h we have f ( *). In order to satisfy both () and (3), f ( *) =. Hence this gives the necessary condition for a relative maima at = * for f ( ). It has to be kept in mind that the above theorem holds good for relative minimum as well. The theorem only considers a domain where the function is continuous and differentiable. It cannot indicate whether a maima or minima eists at a point where the derivative fails to eist. This scenario is shown in Fig 3, where the slopes m 1 and m at the point of a maima are unequal, hence cannot be found as depicted by the theorem by failing for continuity. The theorem also does not consider if the maima or minima occurs at the end point of the interval of definition, owing to the same reason that the function is not continuous, therefore not differentiable at the boundaries. The theorem does not say whether the function will have a maimum or minimum at every point where f () =, since this condition f () = is for stationary points which include inflection points which do not mean a maima or a minima. A point of inflection is shown already in Fig.1 ML1
Optimization Methods: Optimization using Calculus-Stationary Points 4 f() f(*) m 1 m a * Fig. 3 b Sufficient condition: For the same function stated above let f ( * ) = f ( * ) =... = f (n-1) ( * ) =, but f (n) ( * ), then it can be said that f ( * ) is (a) a minimum value of f () if f (n) ( * ) > and n is even; (b) a maimum value of f () if f (n) ( * ) < and n is even; (c) neither a maimum or a minimum if n is odd. Proof Applying the Taylor s theorem with remainder after n terms, we have n 1 n h h ( n 1) h n f ( * + h) = f( *) + hf '( *) + f ''( *) +... + f ( *) + f ( * + θh)! ( n 1)! n! (4) for <θ <1 since f ( * ) = f ( * ) =... = f (n-1) ( * ) =, (4) becomes n h n f ( * + h) f( *) = f ( * + θh) (5) n! As f (n) ( * ), there eists an interval around * for every point of which the nth derivative f (n) () has the same sign, viz., that of f (n) ( * ). Thus for every point * + h of this interval, f (n) ( * + h) has the sign of f (n) ( * ). When n is even n h is positive irrespective of the sign of h, n! and hence f ( * + h) f( *) will have the same sign as that of f (n) ( * ). Thus * will be a relative minimum if f (n) ( * ) is positive, with f() conve around *, and a relative maimum if ML1
Optimization Methods: Optimization using Calculus-Stationary Points 5 f (n) ( * ) is negative, with f() concave around *. When n is odd, n h changes sign with the n! change in the sign of h and hence the point * is neither a maimum nor a minimum. In this case the point * is called a point of inflection. Eample 1. Find the optimum value of the function f ( ) = + 3 5 and also state if the function attains a maimum or a minimum. Solution f '( ) = + 3 = for maima or minima. or * = -3/ f ''( *) = which is positive hence the point * = -3/ is a point of minima and the function attains a minimum value of -9/4 at this point. Eample. Find the optimum value of the function maimum or a minimum. f( ) = ( ) 4 and also state if the function attains a Solution 3 f '( ) = 4( ) = for maima or minima. or = * = for maima or minima. f ''( *) = 1( * ) = at * = f '''( *) = 4( * ) = at * = ML1
Optimization Methods: Optimization using Calculus-Stationary Points 6 * ( ) = 4 f at * = Hence f n () is positive and n is even hence the point = * = is a point of minimum and the function attains a minimum value of at this point. Eample 3. Analyze the function f 5 4 3 ( ) = 1 45 + 4 + 5 and classify the stationary points as maima, minima and points of inflection. Solution f = + = 4 3 '( ) 6 18 1 4 3 => + = or =,1, 3 Consider the point = * = '' * * 3 * * f ( ) = 4( ) 54( ) + 4 = at * = ''' * * * f ( ) = 7( ) 18 + 4 = 4 at * = Since the third derivative is non-zero, = * = is neither a point of maimum or minimum but it is a point of inflection. Consider = * = 1 '' * * 3 * * f ( ) = 4( ) 54( ) + 4 = 6 at * = 1 Since the second derivative is negative the point = * = 1 is a point of local maima with a maimum value of f() = 1 45 + 4 + 5 = 1 ML1
Optimization Methods: Optimization using Calculus-Stationary Points 7 Consider = * = '' * * 3 * * f ( ) = 4( ) 54( ) + 4 = 4 at * = Since the second derivative is positive, the point = * = is a point of local minima with a minimum value of f() = -11 Eample 4. The horse power generated by a Pelton wheel is proportional to u(v-u) where u is the velocity of the wheel, which is variable and v is the velocity of the jet which is fied. Show that the efficiency of the Pelton wheel will be maimum at u = v/. Solution f = K. uv ( u) f = => Kv Ku = u v or u = where K is a proportionality constant (assumed positive). f u v u= = K which is negative. Hence, f is maimum at v u = Functions of two variables This concept may be easily etended to functions of multiple variables. Functions of two variables are best illustrated by contour maps, analogous to geographical maps. A contour is a line representing a constant value of f() as shown in Fig.4. From this we can identify maima, minima and points of inflection. ML1
Optimization Methods: Optimization using Calculus-Stationary Points 8 Necessary conditions As can be seen in Fig. 4 and 5, perturbations from points of local minima in any direction result in an increase in the response function f(), i.e. the slope of the function is zero at this point of local minima. Similarly, at maima and points of inflection as the slope is zero, the first derivatives of the function with respect to the variables are zero. f Which gives us f = ; = at the stationary points, i.e., the gradient vector of f(x), 1 at X = X * = [ 1, ] defined as follows, must equal zero: Δ f This is the necessary condition. f f ( Χ*) f ( Χ*) 1 Δ = = 1 Fig. 4 ML1
Optimization Methods: Optimization using Calculus-Stationary Points 9 Global maima Relative maima Relative minima Global minima Fig. 5 Sufficient conditions Consider the following second order derivatives: f f f ; ; 1 1 The Hessian matri defined by H is made using the above second order derivatives. ML1
Optimization Methods: Optimization using Calculus-Stationary Points 1 H f f 1 1 = f f 1 [ 1, ] a) If H is positive definite then the point X = [ 1, ] is a point of local minima. b) If H is negative definite then the point X = [ 1, ] is a point of local maima. c) If H is neither then the point X = [ 1, ] is neither a point of maima nor minima. A square matri is positive definite if all its eigen values are positive and it is negative definite if all its eigen values are negative. If some of the eigen values are positive and some negative then the matri is neither positive definite or negative definite. To calculate the eigen values λ of a square matri then the following equation is solved. A λi = The above rules give the sufficient conditions for the optimization problem of two variables. Optimization of multiple variable problems will be discussed in detail in lecture notes 3 (Module ). Eample 5. Locate the stationary points of f(x) and classify them as relative maima, relative minima or neither based on the rules discussed in the lecture. f(x) = + 5 ML1
Optimization Methods: Optimization using Calculus-Stationary Points 11 Solution From f (X) =, 1 = + From f (X) = 1 8 + 14 + 3= ( + 3)(4 + 1) = = 3/ or = 1/ 4 so the two stationary points are X 1 = [-1,-3/] and X = [3/,-1/4] The Hessian of f(x) is f f f f = 4 ; = 4; = = 1 1 1 1 ML1
Optimization Methods: Optimization using Calculus-Stationary Points 1 H 4 4 1 = λi-h λ 41 = λ 4 At X 1 = [-1,-3/], λ + 4 λi-h = = ( λ+ 4)( λ 4) 4= λ 4 λ = 16 4 λ = 1 λ =+ 1 λ = 1 1 Since one eigen value is positive and one negative, X 1 is neither a relative maimum nor a relative minimum. At X = [3/,-1/4] λ 6 λi-h = = ( λ 6)( λ 4) 4= λ 4 λ 1λ+ = λ = 5 + 5 λ = 5 5 1 Since both the eigen values are positive, X is a local minimum. Minimum value of f() is -.375. ML1
Optimization Methods: Optimization using Calculus-Stationary Points 13 Eample 6 The ultimate strength attained by concrete is found to be based on a certain empirical relationship between the ratios of cement and concrete used. Our objective is to maimize strength attained by hardened concrete, given by f(x) = and are variables based on cement and concrete ratios. + + 6 3 /, where 1 1 1 Solution Given f(x) = + + 6 3 / ; where X = 1 1 [, ] 1 The gradient vector f ( Χ*) 6 3, to determine stationary point X *. ( Χ*) 1 1 Δ f = = = f Solving we get X * = [1,] f f f = ; = 3; = 1 1 H = 3 λ + λi-h = = ( λ+ )( λ + 3) = λ + 3 Here the values of λ do not depend on X and λ 1 = -, λ = -3. Since both the eigen values are negative, f(x) is concave and the required ratio 1 : = 1: with a global maimum strength of f(x) = 7 units. ML1