REVIEW OF THE SPACE MAPPING APPROACH TO ENGINEERING OPTIMIZATION AND MODELING

Transcription

1 REVIEW OF THE SPACE MAPPING APPROACH TO ENGINEERING OPTIMIZATION AND MODELING Mohamed H. Bakr Simulation Optimization Systems Researh Laboratory and the Department o Eletrial and Computer Engineering, MMaster University, Hamilton, Ontario, Canada, L8S 4K1 John W. Bandler Simulation Optimization Systems Researh Laboratory and the Department o Eletrial and Computer Engineering, MMaster University, Hamilton, Ontario, Canada, L8S 4K1 Kaj Madsen Department o Mathematial Modelling, Tehnial University o Denmark, DK-2800 Lyngby, Denmark Jaob Søndergaard Department o Mathematial Modelling, Tehnial University o Denmark, DK-2800 Lyngby, Denmark Abstrat We review the Spae Mapping (SM) onept and its appliations in engineering optimization and modeling. The aim o SM is to avoid omputationally epensive alulations enountered in simulating an engineering system. The eistene o less aurate but ast physially-based models is eploited. SM drives the optimization iterates o the time-intensive model using the ast model. Several algorithms have been developed or SM optimization, inluding the original SM algorithm, Aggressive Spae Mapping (ASM), Trust Region Aggressive Spae Mapping (TRASM) and Hybrid Aggressive Spae Mapping (HASM). An essential subproblem o any SM based optimization algorithm is parameter etration. The uniqueness o this optimization subproblem has been ruial to the suess o SM optimization. Dierent approahes to enhane the uniqueness are reviewed. We also disuss new developments in Spae Mapping-based Modeling (SMM). These inlude Spae Derivative Mapping (SDM), Generalized Spae Mapping (GSM) and Spae Mapping-based Neuromodeling (SMN). Finally, we address open points or researh and uture development. Keywords: Spae Mapping, Optimization algorithms, Filter Design, Parameter Etration 1. INTRODUCTION We review the Spae Mapping (SM) approah (Bandler et al., ; Bakr et al., ) to engineering devie and system optimization and modeling. The target o system optimization is to determine a set o values or the system parameters suh that ertain design speiiations are satisied. These speiiations represent onstraints on the system responses. Usually, a model o the physial system is utilized in simulating and thus optimizing the system. Traditional optimization tehniques (Bandler and Chen, 1988a) diretly utilize the simulated system responses and possibly available derivatives. Engineering models used in simulating the system 1

2 responses vary in auray and speed. Usually, aurate models are omputationally epensive and less aurate models are ast. In some engineering problems, applying traditional optimization using the aurate models diretly may be prohibitively impratial. On the other hand, applying optimization using the less aurate models may indiate easibility o the design but ould lead to unreliable results. These results must be validated using the aurate models or even using measurements. It ollows that alternative optimization approahes are desirable. SM establishes a mathematial link (mapping) between the spaes o the parameters o two dierent models o the same physial system. The aurate and time-intensive model is denoted as a ine model. The less aurate but ast model is denoted as a oarse model. For eample, in the ontet o analog eletrial iruit design, a ine model may be a time-intensive inite element solution o Mawell equations while the oarse model may be a iruit-theoreti model with empirial algebrai ormulas. Clear distintion should be made between SM optimization and optimization using approimations suh as polynomials, response suraes or splines. All these methods establish a loal approimation o the ine model responses using a set o ine model simulations. This approimation may be updated using new ine model points. On the other hand, SM eploits a oarse model that is physially based and apable o simulating the onsidered system or a wide range o parameter values. This physial model is not updated or hanged during SM optimization. All the SM-based optimization algorithms we will review utilize two steps. The irst step optimizes the design parameters o the oarse model to satisy the original design speiiations. The seond step establishes a mapping between the parameter spaes o the two models. The spae-mapped design is then taken as the mapped image o the optimal oarse model design. Parameter etration is an important element in establishing the mapping. In this step, the oarse model parameters orresponding to a given ine model point are obtained. The etration problem is essentially an optimization problem, and an lead to nonunique solutions. 2

3 The irst SM-based optimization algorithm was introdued in Bandler et al. (1994b). This method assumes a linear mapping between the parameter spaes. This assumption may not be ulilled i signiiant misalignment eists between the two spaes. Here, misalignment denotes the dierene between the ine model response and the oarse model response or the same set o parameters. For two idential models there is no misalignment. The more the untional behaviours o the two models dier, the more the misalignment inreases. Aggressive Spae Mapping (ASM) (Bandler et al., 1995b) eliminates the simulation overhead required in Bandler et al. (1994b). It eploits a quasi-newton step in prediting the new iterates. The algorithm does not assume that the mapping is neessarily linear. However, the nonuniqueness o the parameter etration step may lead to divergene or osillations o the proess (Bandler et al., 1996). Several approahes were suggested to improve the uniqueness o the etration step in the ASM algorithm. These inlude Multi-Point Etration (MPE) (Bandler et al., 1996), the penalty approah (Bandler et al., 1997a) and the statistial parameter etration approah (Bandler et al., 1997b). The Aggressive Parameter Etration (APE) algorithm (Bakr et al., 1999b) addresses the seletion o perturbations utilized in the MPE proess. APE lassiies the possible solutions to the etration problem. The perturbations are obtained by either solving a linear system o equations or through an eigenvalue problem. Trust Region Aggressive Spae Mapping (TRASM) (Bakr et al., 1998; Bandler et al., 1999) integrates a trust region methodology with the ASM tehnique. It also eploits a Reursive Multi-Point Etration (RMPE) proedure. The available inormation about the mapping between the two spaes is utilized in the RMPE. Both ASM and TRASM assume the eistene o a oarse model that has suiient auray. In both algorithms oarse model simulations are used to guide the optimization iterates. I the oarse model is severely dierent rom the ine model both algorithms are not likely to onverge. 3

4 The Hybrid Aggressive Spae Mapping (HASM) algorithm (Bakr et al., 1999; Bandler et al., 1999b) is designed to handle severely misaligned ases. The algorithm utilizes SM optimization as long as SM is onverging. Otherwise, it swithes to diret optimization. The reviewed SM optimization algorithms atually automate and are onsistent with traditional engineering pratie. The rapid development o SM algorithms has not been aompanied, however, with orresponding theoretial development o onvergene properties. A omprehensive theory has yet to be developed. Several approahes have been proposed to utilize the SM onept in engineering modeling. SMbased modeling makes use o both the oarse model and the available mapping between the two spaes. We review three prinipal approahes: Spae Derivative Mapping (SDM) (Bakr et al., 1999a), Generalized Spae Mapping (GSM) (Bandler et al., 1999) and Spae Mapping-based Neuromodeling (SMN) (Bandler et al., 1999d, 1999e). We start by reviewing some onepts and deinitions relevant to engineering devie and system optimization in Setion 2. The basi onept o SM optimization in disussed in Setion 3. The original SM optimization algorithm is disussed in Setion 4. Setion 5 addresses the ASM optimization algorithm along with two variant algorithms. Dierent approahes or improving the uniqueness o the parameter etration proedure are also reviewed in Setion 5. TRASM and HASM are disussed in Setions 6 and 7, respetively. We also give a brie review o reent developments in SM-based modeling approahes in Setion 8. Open points o researh in SM are disussed in Setion 9. Finally, the onlusions are given in Setion SYSTEM OPTIMIZATION: SOME CONCEPTS AND DEFINITIONS The physial system under onsideration an be an eletrial network, an eletroni devie, and so on. The perormane o the system is desribed in terms o some measurable quantities. We denote 4

5 these measurable quantities as the system response untions. The response untions depend on a set o system variables. We utilize the engineering notation system parameters to denote these variables. The response untions are manipulated by hanging some o the system parameters. In other words, these parameters are seleted as optimization variables. We denote these variables as the designable parameters. For eample, the eletrial response o a mirostrip line an be adjusted by hanging the physial width and length o the strip. Usually, some or all physial parameters are seleted as designable parameters and thus an be optimized. We denote the vetor o designable parameters by R n. Eah response untion relies also on some other variables, suh as requeny, time and temperature. These variables are not usually seleted as optimizable parameters. Here, we utilize the engineering notation, independent parameters (Bandler and Chen, 1988a) to denote these variables. i We denote the ith response untion by R (, ξ ), i=1, 2,, N R, where ξ i is the vetor o assoiated i independent parameters. The desired perormane o the system is epressed by a set o speiiations. These speiiations represent onstraints on the responses. They are untions o a set o the independent parameters. In pratie, only a suitable disrete set o samples o the independent parameters is onsidered (Bandler and Chen, 1988a; Bandler and Rizk, 1979). Satisying the speiiations at these sampled values typially implies satisying them or other values o the independent parameters. Let µ i be the number o disrete samples o the ith response. We deine R R m as the vetor o sampled response untions. The kth omponent o R is given by j i R k = R (, ξi ) (1) where k i 1 = µ p p= 1 + j or i = 1, 2,, N R and j=1, 2,, µ i. Here ξ j i is the jth sample o ξ i and m is the total number o sampled response untions. 5

6 An error untion deines the dierene between the speiiation and the orresponding response. In some problems the speiiations deine a target response that should be reahed. These types o speiiations are denoted as single speiiations (Bandler and Chen, 1988a). In other problems, speiiations deine upper and lower bounds on the respetive response. For the ase o single speiiations the error untions are given by where S k is the kth speiiation, k K = {,, } 2 e k = w R S (2) s k k, k k 1 k N k, the set o indies or the onstrained responses, w k is a nonnegative weight and N is the number o speiiations. In the ase o upper and lower speiiations, we lassiy the onstraints on the response untions. We denote by S uk and S lk the kth upper and lower speiiation, respetively. Here, the error untions are given by and e e k k ( ) = w R S, k K u (3) uk k uk ( ) = wlk Slk R, k K l (4) where K u and K l are sets o indies or the onstrained responses and w uk and w lk are nonnegative weights. It is worth mentioning that simultaneous upper and lower speiiations an be imposed on the same sampled response untion, i.e., K u and K l may not be disjoint. Here K u + K l =N. Notie that the symbol denotes the ardinality when applied to a set. Otherwise, it denotes the absolute value. We also denote by e the vetor whose omponents are the error untions given by (2) or by (3) and (4). It is lear rom (3) and (4) that upper and lower speiiations are meaningul only in the ase o a real response while (2) is valid in general or omple responses. Also, a positive, negative or zero value o an error untion indiates that the orresponding speiiation is violated, eeeded or just satisied, respetively. A set o designable parameters or whih e is nonpositive is denoted a easible k 6

7 design. The set o all easible designs deines a easible region in the spae o designable parameters. Fig. 1 illustrates the onepts o error untions, easible design and easible region. The error vetor e is evaluated or a given using the vetor o sampled responses R. R may be obtained by measuring the system responses. However, this approah is usually epensive and time onsuming. Alternatively, R may be obtained by using a model o the physial system. This model utilizes the knowledge available about the physial proesses taking plae within the system. Usually, dierent models eist or the same system. These models vary in their auray and the speed with whih R is obtained. In the disussion that ollows we assume that the system responses are obtained through simulation. The problem o system design an be ormulated as * = arg minu ( ) (5) where U() is a salar objetive untion that is dependent on the error untions. U() should oer a measure o the speiiations violation or satisation. A possible hoie o U() is the l p norm (Temes and Zai, 1969), Huber norm (Ekblom and Madsen, 1989; Huber, 1981) or the generalized l p untion (Bandler and Charalambous, 1972; Charalambous, 1977). The l p norm o e is given by 1/ p N p e p = ek (6) k= 1 The most ommonly used norm is the l2 norm, i.e., p=2. This norm is widely used beause o its dierentiability and its statistial properties. A large number o optimization tehniques eist or leastsquares optimization (Dennis, Jr., and Shnabel, 1983). Solutions obtained using least-squares optimization an be altered signiiantly by the eistene o a ew wild data points. Setting p=1 we have the l1 norm N ek k= 1 e = (7) 1 7

8 This norm is robust to outliers. It inds wide appliation in data-itting in the presene o gross errors (Bandler et al., 1987), in analog ault loation (Bandler and Salama, 1985b) and devie modeling (Bandler et al., 1986a). Setting p= we have the l norm e = ma ek (8) k whih onsiders only the worst violated error untion. Many system design problems an be ormulated as a minima optimization problem (Bandler et al., 1985a, 1986b; Charalambous, 1974; Madsen et al., 1975b). The l1 and l norms are both nondierentiable. Corresponding optimization algorithms tend to be more involved than least-squares algorithms. In general, the algorithms used to minimize the l1 and l norms ollow similar strategies. These algorithms solve the minimization problem in an iterative way. Murray and Overton (1981) and Waren et al. (1967) ormulated the problem as a nonlinear program. Some methods utilize irst-order derivatives o the error untions to onstrut sequential linearizations o the nonlinear program. Suh methods are denoted as irst-order methods. For eample, in Osborne and Watson (1969, 1971) the linearization is used to onstrut a linear program that returns a suggested searh diretion. A line searh is then eeuted in that diretion. A trust region methodology (Moré, 1982) is integrated with the linear program ormulation in Madsen (1975a). Some o these irst-order methods assure global onvergene to a stationary point, or eample (Madsen, 1975a). However, they may yield a low onvergene rate in the neighborhood o a solution i the problem is singular (Madsen and Shjær-Jaobsen, 1976). Another lass o methods or the minimization o l1 and l norms utilizes approimate seondorder derivatives o the error untions (Hald and Madsen, 1981, 1985). These methods solve the irstorder optimality onditions using quasi-newton methods (Broyden, 1967; Davidon, 1959; Dennis, Jr., and Moré, 1977; Dennis, Jr., and Shnabel, 1983). They usually have a high onvergene rate in the 8

9 neighborhood o a solution. However, pure seond-order methods do not guarantee global onvergene. Hybrid methods (Bandler et al., 1985a, 1987) ombine both irst-order and seond-order methods. A irst-order method is used ar rom the solution. One the solution is approahed, a swith to a seondorder method is eeuted. Several swithes an take plae between the two methods. Another norm that an be utilized as an objetive untion is the Huber norm (Ekblom and Madsen, 1989; Huber, 1981). This norm is a hybrid ombination between the l1 and l2 norms. It is deined by where N e = ρ ( e ) (9) H α k= 1 k 2 ek 2 ρα ( ek) = α e k 2 α 2 i i e e k k α > α (10) where α is a threshold alled the Huber threshold. This norm treats small errors in the l2 sense while it treats large errors in the l1 sense. Here, we adopt the engineering notation and use the word norm or the untion (9) even though some o the basi norm properties are not satisied. Huber optimization (Bandler et al., 1993b) is more robust against gross errors than least-squares optimization. It also oers less biased designs than those obtained using l1 optimization (Bandler et al., 1993b). The previously disussed norms, an be used to minimize the error untions towards zero. A design that orresponds to a zero error vetor would be satisatory i it were not or manuaturing toleranes. These toleranes are inevitable and may ause the onstruted physial system to violate the speiiations. It ollows that optimization should ontinue to enter the design within the easible region (Bandler, 1974; Bandler and Abdel-Malek, 1978; Bandler et al., 1976, 1988b). The yield is deined as the perentage o the manuatured systems that satisy the design onstraints. Fig. 2 Illustrates the 9

10 onepts o design entering and yield. Several algorithms have been developed with the aim o maimizing the yield (Abdel-Malek and Bandler, 1980a, 1980b; Bandler et al., 1993a). The generalized l p untion (Bandler and Charalambous, 1972; Charalambous, 1977) was developed to enable optimization towards a better entered design. It makes use o the one-sided objetive untions and + H = p e k p k H p e k 1/ p p ( ) = k, e k 0 (11) 1/ p, e k < 0 (12) The generalized l p untion is equal to (11) i at least one o the speiiations is violated. Otherwise, it is equal to (12). We denote the optimization algorithms disussed thus ar as diret optimization algorithms. They utilize simulations o the optimized system and an be applied i the model simulation time is not etensive. Otherwise, diret optimization beomes prohibitive and alternative methods should be used. SM optimization was introdued as suh an alternative. 3. SPACE MAPPING OPTIMIZATION: THE BASIC CONCEPT Spae Mapping aims at eiiently solving the optimization problem (5). Here the only onstraints present are given by the design speiiations. The ase o linearly onstrainted designable parameters was not addressed. It will be the subjet o a uture researh. We reer to the vetors o ine model parameters and orresponding oarse model parameters as R n and R n, respetively. The optimal oarse model design * is obtained using only oarse model simulations. The orresponding response is denoted by R *. A minima algorithm (Bandler et al., 1985a), i appropriate, may be used. 10

11 suh that SM establishes a mathematial link (mapping) P between the two spaes (Bandler et al., 1994b) = P( ) (13) R ( ) R ( ) ε (14) The mapping P is valid over a region in the parameter spae. An approimation to this mapping is established in an iterative way. We denote by P ( j ) the available approimation to P at the jth iteration. The orresponding ine model design is given by ( j 1 + 1) = P ( * ) (15) I it satisies a ertain termination riterion, it is aepted as the spae mapped design. Otherwise, the mapping is updated and a new design is alulated. 4. THE ORIGINAL SM OPTIMIZATION ALGORITHM ( At the jth iteration, the algorithm utilizes a set o ine model points S j ) deined by S (1) (2) ( m ) {,, j } = (16), where (0) m j = S. The points in S are seleted in the viinity o a reasonable andidate or the ine model design. In (Bandler et al., 1994b) it is suggested to take ( 1). The other m 1 points are = * 0 seleted by perturbation with m ( n 1. The ine model response or every point in the set S j ) is 0 + simulated. A orresponding set o oarse model points S ( j) deined by S (1) (2) ( m ) {,, j } = (17), is then onstruted. The points ( i) ( S j ), i=1, 2,, m j are obtained through the Single-Point Etration (SPE) proess (see Fig. 3) 11

12 ( i) ( i = arg min R ( ) ) R ( ) (18) ( j is then estimated using S j P ) ( ) and S ( j) using (13)-(14). Here, Every oarse model parameter is epressed as a linear ombination o some predeined and ied untions ϕ k ( ), k=0, 1,, l. It ollows that = P ( ) = A ϕ( ) (19) where ( l+ 1) A R n is a matri o onstant oeiients and ϕ( ) is given by ϕ ( ) = ϕ 0 ( ϕ1( ϕl ( ) ) ) (20) ( Relation (19) must be satisied or every pair o orresponding points in S j A should satisy (2) ( [ m ) j ] A (1) = ) and S ( j). It ollows that (1) (2) ( ) [ ϕ( ) ϕ( ) ( m j ) ] (21) ϕ Bandler et al. (1994b) assumed that the mapping between the two spaes is linear, i.e., = P ( ) = B (22) + where ( j R n n and j R B ) ( ) n ( j ). The linear mapping (22) is equivalent to (19) with = [ ] A B k, k and ϕ ( ) =, k=1, 2,, n, the kth omponent o the vetor, ϕ ( ) 1 and l=n. It ollows that (21) an be written as A least-squares solution or A T is thus given by 0 = A (1) (2) ( ) (23) m j (1) (2) ( [ m j) ] = A T T ( D D) D Q = (24) T 1 12

13 where 1 = 1 D (1) (2) ( m ) j 1 T (25) and (1) (2) ( [ m j ] Q ) T One A is obtained, the suggested spae-mapped design is = (26) ( m j 1 ) 1 * ( ) ( ) + 1) * j = P ( = B (27) ( m +1) Here, the mapping is assumed to be one to one. The new point j is taken as an approimation to the optimal ine model design * i the ondition ( m j+ * R ( 1) ) R ( ) ε (28) is satisied. In this ase we take ( +1) = m j (. Otherwise, the set S j ) ( m +1) is augmented by j and the set ( ( m +1) is augmented by j obtained using (18). The algorithm steps using (23)-(28) are then repeated S j ) using the augmented sets. Fig. 4 illustrates one iteration o the algorithm. This algorithm is simple but it suers rom a number o drawbaks. First, to have the algorithm started an initial set o ine model points S (0) must be reated. Simulating S (0) represents a signiiant overhead or the algorithm. The mapping is also assumed to be linear, whih may not be true or signiiantly misaligned models. Also, oarse model points are obtained through SPE. Nonuniqueness o the etrated parameters may lead to an erroneous mapping estimation and divergene o the algorithm. These drawbaks led to the development o the Aggressive Spae Mapping (ASM) algorithm (Bandler et al. 1995b). 5. THE ASM ALGORITHM The spae-mapped design is a solution to the system o nonlinear equations 13

14 * = P( ) = 0 (29) ASM solves (29) in an iterative manner. Let be the jth iterate in the solution o (29). The net ( j+ iterate 1) is ound by a quasi-newton iteration h is a solution o ( j+ 1) = h (30) + B h ( ) = (31) j where * = P ( ). B is an approimation to the Jaobian J m o with respet to at. J m is deined by J m T T ( ) ( ) = (32) = I the mapping between the two spaes is linear, similar to (22), the matri J m is onstant. Otherwise, it is a untion o the ine model parameters. The initial approimation to J m is taken as matri. B is updated at eah iteration using Broyden s rank one update (Broyden, 1965) B (0) = I, the identity B ( j+ 1) ( j + 1) B h T = B + h j) T ( h h (33) The ormula (33) an be simpliied using (31) to B ( j+ 1) ( j + 1) T = B + h T h h (34) The error vetor ( j ) is obtained by evaluating P ( ), whih is done indiretly through SPE. The algorithm terminates i beomes suiiently small. A omplete iteration o the algorithm is shown in Fig. 5. The ASM algorithm does not require an initial set o ine model points. This implies that there is no simulation overhead. Also, while (23) assumes that the mapping is linear, ASM does not make this 14

15 assumption. The output o the ASM algorithm is the spae-mapped design and the matri B, whih approimates the Jaobian J m at. However, the nonuniqueness problem o the SPE proess remains. An inorret value or the vetor P ( ) may ause the algorithm to diverge or ehibit osillatory behavior. Two interesting, intuitive, variants o the ASM algorithm are suggested by Bila et al. (1998) and Pavio (1999). The basi idea o both algorithms is pratially the same. The iterate is given by (31) with the matri B ( j ) ied at B ( j ) =I. Broyden s updating ormula is not utilized. These steepest-desent approahes may sueed i the mapping between the two spaes is essentially represented by a shit. An eample o ASM optimization is the three-setion mirostrip impedane transormer (Bakr et al., 1997). The ilter struture is shown in Fig. 6. The ine model utilizes the ull-wave eletromagneti simulator em (1997). The oarse model utilizes the empirial mirostrip line and mirostrip step models available in the iruit simulator OSA90/hope (1997). The designable parameters are the width and physial length o eah mirostrip line. Here, the reletion oeiient S11 is used to math the two model responses. ASM terminated using only 9 ine model simulations. The initial and spae-mapped responses are shown in Figs. 7 and 8, respetively. A requeny sweep o the ine model requires about one hour o CPU time on an HP workstation model 715/33. The oarse model simulation time is a ration o a seond. Optimizing this iruit using diret methods would have probably required dozens o ine model simulations. Several approahes were suggested to enhane the uniqueness o the parameter etration proess. The irst approah is Multi-Point Etration (MPE) (Bandler et al., 1996). It simultaneously mathes a number o points in both spaes. MPE aims at mathing not only the untion values but also the irst-order derivatives. The point orresponding to is ound by solving j+ 1) T T T T ( = arg min [ e0 e1 e ] Np (35) 15

16 where ( j 1) = R ( ) R ( ) (36) + e 0 and ( i) ( j+ 1) ( i) e = R ( + ) R ( + ), i=1, 2,, N p (37) i ( j+ 1) ( j+ 1) ( i) It ollows that the set o utilized ine model points is V = { } { + i = 1, 2, }, N p. The perturbations (i) and ( ) i are related by (Bandler et al., 1996) ( i) ( i) =, i=1, 2,, N p (38) Integrating this MPE in the ASM algorithm aes some diiulty. The number o ine model points utilized is arbitrary and there is no lear way o how to selet them. Also, available inormation about the mapping between the two spaes is not utilized. This MPE proedure is illustrated in Fig. 9. Another approah is suggested by Bandler et al. (1997a). Here, the point is obtained by solving the penalized SPE proess ( j+ 1) * = arg min ( R ( ) R ( + w ) (39) ( j + 1) ) where w is a weighting ator. I the parameter etration problem is not unique (39) is avored over (18). The solution o the etration problem is biased towards the point * and thus drives the error vetor to zero as the algorithm proeeds. I w is too large the mathing between the responses is poor. On the other hand, too small a value o w makes the penalty term ineetive. In whih ase, the uniqueness o the etration step may not be enhaned. A statistial approah to parameter etration is suggested in (Bandler et al., 1997b). Here, the SPE proess (18) is initiated rom dierent starting points. The etration proess is unique i the same values o the etrated parameters are obtained or eah starting point. Otherwise, the solution is nonunique. In this ase, the solution that results in the best math in terms o some norm is seleted. 16

17 Bandler et al. (1997b) suggested that a set o N s starting points be randomly seleted in a region D R n where the solution is epeted. For the jth iteration, D is deined by ( j ) i i, i [ * i 2, *, i + 2 ] (40), or i=1, 2,, n. Fig. 10 illustrates the seletion o the interval D or the two-dimensional ase. The Aggressive Parameter Etration (APE) algorithm (Bakr et al., 1999b) addresses the seletion o the perturbations utilized in the MPE proess. It suggests perturbations that are likely to impat the uniqueness o the parameter etration step. APE lassiies the possible solutions o the parameter etration problem as either loally nonunique or loally unique. In the loally nonunique ase the minimum o the etration problem is assumed over a surae. For the loally unique ase the minimum is assumed at a point. Figs. 11 and 12 illustrate the lassiiation o the etrated parameters. To illustrate the APE algorithm, assume that the point orresponding to is obtained through MPE. The utilized set o ine model points is V with V = N. I is loally nonunique, APE suggests a new point to be added to V. This point is likely to make the etrated parameters using the augmented set loally unique. It is obtained by solving a linear system o equations that utilizes the gradients and Hessians o oarse model responses at j. ( +1) I is loally unique, the new point (N ) to be added to the set V is obtained by solving the eigenvalue problem and ( ( j 1) T ( j 1) ( ) J J ( + N ( ) ) + I) = λ ( N ) + (41) ( N ) ( j+ 1) ( N ) = + (42) 17

18 where J is the Jaobian o oarse model responses. Here, the oarse model perturbation ( N ) and the ine model perturbation ( N ) are related by the available mapping. The obtained perturbation is saled to satisy a ertain trust region. 6. THE TRASM ALGORITHM TRASM (Bakr et al., 1998; Bandler et al., 1999a) integrates a trust region methodology (Moré, 1982) with the ASM tehnique. Similar to ASM, TRASM aims at solving (29). However, instead o utilizing a quasi-newton step the problem is solved as a least-squares problem. In the jth iteration the objetive o TRASM is to minimize ( + 1) 2 2 j within a ertain trust region. To ahieve this, TRASM ( j +1) utilizes a linearization o the vetor untion. The linearized objetive untion is thus given by The suggested step is obtained by solving h 2 2 ( ) ( ) ( ) h ) j j j L (, = + B h (43) ( ) 2 = arg min + B h (44) 2 h j subjet to h ( j ) δ (45) 2 where δ is the size o the trust region. The solution o (43)-(45) is obtained by solving (Levenberg, 1944; Marquardt, 1963) ( B B T + λ I h = B (46) T ) where λ orrelates to δ. The larger the value o δ the smaller the value o λ and vie versa. TRASM makes use o the algorithm suggested in (Moré and Sorenson, 1983) to determine the value o λ. 18

19 ( j+ 1) + The suggested iterate is = h. Unlike ASM, is aepted only i it satisies a suess riterion with respet to the redution in the 2 norm o the vetor. The suess riterion utilized by TRASM is ( j+ 1) ( k ) ( + B h ) > 0.01 (47) The subsript k indiates the number o points utilized in the Reursive Multi-Point Etration (RMPE). It ollows that k = V, where V is the set o ine model points used in the RMPE. Initially V={ ( j +1) } and k=1. I (47) is satisied is aepted and B is updated using (33). Otherwise, the validity o the ( j +1) ( j +1) etration proess leading to k is suspet. The residual vetor k is then used to onstrut a temporary point (k t ) rom the point ( j+ 1) by using (46). The set V is updated to V (k ) t repeated using the augmented set V to get ( j + 1) k + 1. RMPE is then. RMPE is given by (35)-(37) with (38) replaed by ( i) ( i) = B (48) Thus, the available inormation about the mapping between the two spaes is eploited. Fig. 13 illustrates the RMPE proedure. ( j+ 1) The new error vetor k either satisies (47) or it is used to obtain another additional point + 1 whih is then added to the set V. RMPE is then repeated until the etrated parameters are trusted (see Fig. 14). TRASM trusts the vetor o etrated parameters i it approahes a limit. The suiient ondition or this is ( j + 1) ( j + 1) k + 1 k ε (49) 19

20 The trusted value o k is denoted simply by ( j +1). I satisies (47), is aepted and the matri B is updated using (33). Otherwise, the auray o the linearization used to predit h is suspeted. Thus, to ensure a suessul step rom the urrent point ( j), the trust region size is shrunk and a new suggested point is obtained. During RMPE we may have V = n + 1. In this ase, suiient inormation is available to obtain an estimate or the Jaobian J o the ine model responses. J is then used to obtain an alternative h ( j ). The size o the trust region is updated at the end o eah iteration based on the math between the atual redution and the predited redution in. The trust region size is inreased i the ondition ( j ) ( j+ 1) ( ) 0.80 ( + B h ) (50) is satisied. It should be mentioned that the onstants utilized in (47) and (50) are arbitrary. The design o a High-Temperature Superonduting (HTS) ilter (Bandler et al., 1995a) is arried out using TRASM (Bakr et al., 1998). The ilter is shown in Fig. 15. The designable parameters are L 1, L 2, L 3, S 1, S 2 and S 3. The oarse model eploits the empirial models o a mirostrip line, oupled lines and open stubs available in OSA90/hope. The ine model employs the method o moments simulator em. The initial ine model response is shown in Fig. 16. Only 8 ine model simulations were required by TRASM. The spae-mapped response is shown in Fig. 17. On a Sun SPARCstation 10, the ine model requires one hour o CPU time per requeny point. The oarse model requires a ration o a seond or a omplete sweep. Both the ASM and TRASM algorithms are eiient. The number o required ine model simulations is o the order o the problem dimensionality. However, both models depend on the eistene o a oarse model that is ast and has suiient auray. The main predition steps in (31) and (46) show that oarse model simulations are used to guide the optimization iterates. I the oarse model is severely dierent rom the ine model the ASM algorithm is likely to diverge and TRASM may stop at a 20

21 solution that is not lose to the required design. To overome this problem the Hybrid Aggressive Spae Mapping (HASM) algorithm was developed. 7. THE HASM ALGORITHM HASM eploits SM when eetive, otherwise it deaults to diret optimization. Two objetive untions are utilized by the algorithm. The irst objetive untion is 2 2 ( * 2 whih is the TRASM objetive untion. The seond objetive untion is g 2 2 j) = P ( ) (51) 2 2 * = R ( ) R ( ) (52) and is denoted as the diret optimization objetive untion. HASM onsists o two phases: the irst phase ollows the TRASM strategy while the seond phase eploits diret optimization. For swithing between the two phases the algorithm utilizes a relationship that relates the established mapping to the irst-order derivatives o both models (Bakr et al., 1999; Bandler et al., 1999b]. This relationship stipulates that i orresponds to through a 2 parameter etration proess, then the Jaobian J o the ine model response at and the Jaobian J o the oarse model response at are related by J = J B (53) where B is a valid mapping between the two spaes at and. Another important relationship that ollows rom (53) is T -1 T ( ) J J B = J J (54) (54) assumes that J is ull rank and m n, where m is the dimensionality o both R and R. 21

22 ( j ) In the jth iteration we assume the eistene o a trusted = P ( ). The step taken is given ( j ) ( j+ 1) by (46) where = h. + ( j+ 1) * 1 ( j+ 1) SPE is then applied at to get = P( ). The irst phase utilizes two suess riteria related to the redution in (51) and (52). The irst suess riterion is given by (47). The suess riterion related to (52) is given by g ( j+ 1) < g (55) is aepted i (55) is satisied. The irst phase ontinues and the matri B is updated i ( j +1) (47) is also satisied or a trusted. Swithing to the seond phase takes plae in two ases. The irst ase ours i (55) is not satisied. The seond phase is then supplied by j ) (, J and. Here, J is estimated rom J and B by using (53). The seond ase ours when satisies (55) but does not satisy (47) or a trusted. ( j is updated to B ( j+ 1) B ) using (33). The seond phase is supplied with, and J. J is estimated rom J and B ( +1) j by using (53). I V reahes n+1 during RMPE, J is instead estimated through inite dierenes. The seond phase utilizes the irst-order derivatives supplied by the irst phase to arry out a number o suessul iterations with the target o minimizing (52). At the end o eah suessul iteration ( +1) k parameter etration is applied at the new iterate and is used to hek whether a swith to the irst phase is possible. Here, k is used as an inde or the iterations o the seond phase. In the original implementation, swithing bak to the irst phase takes plae i ( k k < + 1) ( ) (56) 22

23 ( k+1) ( k+ 1) ( k+ 1) In this ase J is evaluated at P( ) =. B is then reovered using (54). Fig. 18 illustrates the onnetion between SM optimization and diret optimization. HASM is illustrated by onsidering a si-setion H-plane waveguide ilter (Matthaei et al., 1964; Young and Shiman, 1963). The ilter is shown in Fig. 19. The ine model utilizes the inite element simulator HP HFSS (1998) through HP Empipe3D (1998). The designable parameters are the our septa widths W 1, W 2, W 3 and W 4 and the three waveguide-setion lengths L 1, L 2 and L 3. The oarse model onsists o lumped indutanes and dispersive transmission line setions. It is simulated using OSA90/hope. A simpliied version o a ormula due to Maruvitz (1951) is utilized in evaluating the indutanes. The oarse model is shown in Fig. 20. The responses obtained through dierent design stages are shown in Figs. 21 and 22. In a later implementation, both the reovery o B and the swithing bak riterion were modiied. The mapping reovery step (54) is made better onditioned by onstraining B to be lose to the identity matri I. This ollows rom the at that the ine and oarse models share the same physial bakground (Bandler et al., 1999). B is thus obtained by solving T T T T T T T 2 B = arg min [ e1 e2 en w b1 w b 2 w b n ] (57) 2 B where e i is the ith olumn o the matri ( k + 1) ( k+ 1) ( k+ 1) E = J J B (58) b i is the ith olumn o the matri B = B I (59) The solution to (57) is given by B ( k+ 1) T ( k+ 1) 2 1 ( k+ 1) T ( k 1) 2 ( + w I) ( J J + w I) + = J J (60) (60) is idential to (54) i w=0. In this modiied implementation swithing bak to the irst phase takes plae i B given by (60) is able to predit with suiient auray the redution in (k). 23

24 8. SM-BASED MODELING Thus ar we oused on optimization algorithms. In this setion we briely disuss some o the SM-based modeling algorithms. The basi onept is to establish a mapping between the parameter spaes that is given by (13) and (14). The ine model response is then approimated by R ( ) R ( P( )) (61) The model given by (61) oers a ast approimation to the time-intensive ine model response. SMbased modeling approahes dier in the way in whih the mapping is established, the nature o the mapping and the region o validity o the obtained model. We review three o these algorithms; Spae Derivative Mapping (SDM), Generalized Spae Mapping (GSM) and Spae Mapping-based Neuromodeling (SMN). Fig. 23 illustrates the onept o SM based modeling. Spae Derivative Mapping (SDM) (Bakr et al., 1999a) This algorithm develops a loally valid approimation o the ine model in the viinity o a partiular point. We denote by J the Jaobian o the ine model responses at. The irst step o the algorithm is to obtain the point orresponding to through the SPE problem (18). The Jaobian J at may be estimated by inite dierenes. Both (18) and the evaluation o J should add no signiiant overhead. The mapping matri B is then alulated by applying (54) as One B is available the linear mapping is given by T 1 T ( ) J J B = J J (62) = P( ) = + B ( ) (63) The SDM model is given by (61) with P given by (63). In a later implementation, the matri B is estimated using (60). 24

25 Generalized Spae Mapping (GSM) Modeling (Bandler et al., 1999) This approah integrates three previously suggested SM modeling onepts (Bandler et al. 1994b, 1995b, 1998). The model is epeted to be aurate in a region o the ine model spae D R n. The mapping between the two spaes is assumed to be o the orm P( = B + (64) = ) A set o ine model points V D is onstruted. The mapping parameters B and are then obtained through the optimization proedure where T T T T B, ] = arg min [ e1 e en ] (65) B, [ 2 ( i) i ( i) e = R ( B + ) R ( ) (66) where ( i V, i=1, 2,, N and V = N. A star-like set o points is utilized in Bandler et al. (1999). ) This seletion o V is illustrated in Fig. 24 or the three-dimensional ase. In (65) B an be onstrained to be lose to I similar to (60). Another variation o (65) that is pertinent to analog eletrial iruit devie modeling is to inlude the requeny as a mapped parameter. This is essential i there are onstraints on the possible simulated requenies o the oarse model (Bandler et al., 1999d). Also, it is reported that the auray o the SM model is signiiantly improved by utilizing a requeny-sensitive mapping. Spae Mapping-based Neuromodeling (SMN) (Bandler et al., 1999d, 1999e) In Setion 4 we notied that the basi idea o the original SM optimization algorithm is to epress eah oarse model parameter as the sum o predeined untions o the ine model parameters. Relation (19) an be written as n j ), i j= 1 = aijϕ ( (67) 25

26 Consider a three-layer Artiiial Neural Network (ANN) (Burrasano and Mongiardo, 1999). The inputs to this network are the ine model parameters and the outputs approimate the orresponding oarse model parameters. It ollows that eah output an be epressed as (see Fig. 25) i nh T y = aij ψ ( w j + θ j) (68) j= 1 where n h is the number o hidden neurons and ψ j is the ativation untion assoiated with the jth hidden T nh neuron. Here, = [ ai1 ai2 ai ] i j a is the vetor o weights assoiated with the ith output neuron, w j is the vetor o weights assoiated with the jth hidden neuron and θ j is the orresponding threshold. By omparing (67) and (68) we see that a trained ANN an approimate the mapping between the two spaes. The universal approimation theorem (Burrasano and Mongiardo, 1999) assures that a three-layer ANN is apable o approimating any nonlinear mapping between the two spaes. Similar to GSM, an ANN is trained to approimate the mapping between the two spaes in a subset D o the parameter spae. Given a set o training points V D, the training problem is given by where and i=1, 2,, N and T T T T W, θ, A] = arg min [ e1 e e N ] (69) W, θ, A [ 2 V = N. ( i) e = R ( y) R ( ) (70) i The optimized mapping parameters are deined by W T = [ w1 w2 wnh], θ = [ θ1 θ2 θ ] and A [ a1 a2 an] nh = (71) This approah is superior to other modeling approahes that utilize ANNs (Watson and Gupta, 1996; Zaabab et al., 1995) beause it results in a simple neural network and utilizes ewer training points (Bandler et al., 1999d). 26

27 Several variations to this approah that are more pertinent to analog eletrial iruit modeling have been suggested by Bandler et al. (1999d, 1999e). The main onept in all these variations is to obtain a requeny-sensitive mapping that improves the auray o the SM model. The star distribution shown in Fig. 24 is also used or V. The GSM approah an be visualized as a speial ase o SMN where the ANN has only two layers with no hidden neurons (see Fig. 26). 9. FUTURE RESEARCH IN SM Coarse Model Generation All the SM-based algorithms thus ar depend on the eistene o a oarse model with suiient auray. The generation o suh a model requires knowledge o the problem and is still the user s responsibility. We epet more researh on the automated generation o ast oarse models that have suiient auray. Lightly trained neural networks may be one possible solution to this problem. Neural Network-Based SM Optimization Reently, an SM neuromodeling approah was introdued (Bandler et al., 1999d, 1999e). A pioneering work (Bakr et al. 2000a) etends this onept to SM optimization. Here, a SM Neuromodel is utilized in optimizing the ine model. The ompleity o the ANN is inreased in every iteration with the newly generated ine model points. We epet urther researh to be arried out in this diretion. Optimality Conditions o SM Optimization Researh is being arried out to develop a omprehensive theory or the optimality onditions o SM. Development o suh a theory will help robustize the SM-based optimization algorithms. Using Surrogate Models Researh is urrently onduted on the integration o surrogate models (Aleandrov et al., 1998; Booker et al., 1999; Dennis, Jr., and Torzon, 1995; Torzon and Trosset, 1998; Trosset and Torzon, 1997) with SM optimization. Surrogate-based optimization aims at eiiently optimizing a omputationally-epensive model. Unlike SM optimization, the design problem is not ormulated as an 27

28 equivalent nonlinear system. Alternatively, the original design problem is solved using an approimate model. This approimate model may be a less aurate physially-based model or an algebrai model. The generated iterates are validated through ine model simulations. The auray o the surrogate model is improved in every iteration using the generated simulations. A novel work (Bakr et al., 2000b) ombines SM and surrogate model optimization in a powerul algorithm. We epet that more researh will be onduted in this area. 10. CONCLUSIONS In this work we reviewed the SM approahes to engineering optimization and modeling. SM optimization makes use o the eistene o a less aurate but ast model to aelerate the optimization problem. The algorithms reviewed inlude the original SM optimization algorithm, ASM, TRASM and HASM algorithms. The original SM optimization algorithm utilizes two orresponding sets o points to establish the mapping between the two spaes. ASM eliminates the overhead simulations to obtain the initial mapping. However, it suers rom the nonuniqueness o the parameter etration subproblem. The TRASM algorithm integrates a trust region methodology with the ASM tehnique. It also utilizes a reursive multi-point etration approah. HASM addresses the problem o severely misaligned oarse models. It allows swithing between SM optimization and diret optimization. We also reviewed the dierent approahes or improving the uniqueness o the parameter etration problem. These inlude multi-point etration, penalized parameter etration, statistial parameter etration and aggressive parameter etration. The dierent approahes or SM-based engineering modeling were briely disussed. We reviewed the SDM, the GSM and the SMN algorithms. Finally, we suggested some o the open points or researh in SM. 28

29 ACKNOWLEDGEMENT The authors thank Sonnet Sotware, In., Liverpool, NY or making em available or this work. They also thank Agilent Tehnologies, Santa Rosa, CA, or making HP HFSS and HP Empipe3D available. The authors would also like to thank their olleagues Dr. N. Georgieva, M.A. Ismail, J.E. Rayas-Sánhez and Dr. Q.J. Zhang (Carleton University) or useul disussions that helped shape our work. Thanks are also due to the three reerees and Dr. T. Terlaky or their useul suggestions and omments. This work was supported in part by the Natural Sienes and Engineering Researh Counil o Canada under Grants OGP , STP , through the Mironet Network o Centres o Eellene and Bandler Corporation. M.H. Bakr is supported by an Ontario Graduate Sholarship. REFERENCES H.L. Abdel-Malek, and J.W. Bandler, Yield optimization or arbitrary statistial distributions, Part I: Theory, IEEE Trans. Ciruits Syst., vol. CAS-27, pp , 1980a. H.L. Abdel-Malek, and J.W. Bandler, Yield optimization or arbitrary statistial distributions, Part II: Implementation, IEEE Trans. Ciruits Syst., vol. CAS-27, pp , 1980b. N. Aleandrov, J.E. Dennis, Jr., R.M. Lewis and V. Torzon, A trust region ramework or managing the use o approimation models in optimization, Strutural Optimization, vol. 15, pp , M.H. Bakr, J.W. Bandler, R.M. Biernaki and S.H. Chen, Design o a three-setion 3:1 mirostrip transormer using aggressive spae mapping, Report SOS-97-1-R, Simulation Optimization Systems Researh Laboratory, MMaster University, Hamilton, Canada, M.H. Bakr, J.W. Bandler, R.M. Biernaki, S.H. Chen and K. Madsen, A trust region aggressive spae mapping algorithm or EM optimization, IEEE Trans. Mirowave Theory Teh., vol. 46, pp , M.H. Bakr, J.W. Bandler and N. Georgieva, Modeling o mirowave iruits eploiting spae derivative mapping, IEEE MTT-S Int. Mirowave Symp. Dig. (Anaheim, CA), pp , 1999a. M.H. Bakr, J.W. Bandler and N. Georgieva, An aggressive approah to parameter etration, IEEE Trans. Mirowave Theory Teh., vol. 47, pp , 1999b. 29

30 M.H. Bakr, J.W. Bandler, N. Georgieva and K. Madsen, A hybrid aggressive spae mapping algorithm or EM optimization, IEEE Trans. Mirowave Theory Teh., vol. 47, pp , M.H. Bakr, J.W. Bandler, M.A. Ismail, J.E. Rayas-Sánhez and Q.J., Zhang, Neural spae mapping optimization o EM mirowave strutures, IEEE MTT-S Int. Mirowave Symp. Dig. (Boston, MA), pp , 2000a. M.H. Bakr, J.W. Bandler, K. Madsen, J.E. Rayas-Sánhez and J. Søndergaard, Spae mapping optimization o mirowave iruits eploiting surrogate models, IEEE MTT-S Int. Mirowave Symp. Dig. (Boston, MA), pp , 2000b. J.W. Bandler, Optimization o design toleranes using nonlinear programming, J. Optimization Theory and Appliations, vol. 14, pp , J.W. Bandler, and H.L. Abdel-Malek, Optimal entering, toleraning and yield determination via updated approimations and uts, IEEE Trans. Ciruits Syst., vol. CAS-25, pp , J.W. Bandler, M.H. Bakr, N. Georgieva, M.A. Ismail and D.G. Swanson., Jr., Reent results in eletromagneti optimization o mirowave omponents, inluding mirostrip T-juntions, Pro. 15th Annual Review o Progress in Applied Computational Eletromagnetis ACES 99 (Monterey, CA), pp , 1999a. J.W. Bandler, M.H. Bakr, and J.E. Rayas-Sánhez, Aelerated optimization o mied EM/iruit strutures, Pro. Workshop on Advanes in Mied Eletromagneti Field and Ciruit Simulation, IEEE MTT-S Int. Mirowave Symp. (Anaheim, CA), 1999b. J.W. Bandler, R.M. Biernaki and S.H. Chen, Fully automated spae mapping optimization o 3D strutures, IEEE MTT-S Int. Mirowave Symp. Dig. (San Franiso, CA), pp , J.W. Bandler, R.M. Biernaki, S.H. Chen, W.J. Gestinger, P.A. Grobelny, C. Moskowitz and S.H. Talisa, Eletromagneti design o high-temperature superonduting ilters, Int. J. Mirowave and Millimeter- Wave Computer-Aided Engineering, vol. 5, pp , 1995a. J.W. Bandler, R.M. Biernaki, S.H. Chen, P.A. Grobelny and R.H. Hemmers, Eploitation o oarse grid or eletromagneti optimization, IEEE MTT-S Int. Mirowave Symp. Dig. (San Diego, CA), pp , 1994a. J.W. Bandler, R.M. Biernaki, S.H. Chen, P.A. Grobelny and R.H. Hemmers, Spae mapping tehnique or eletromagneti optimization, IEEE Trans. Mirowave Theory Teh., vol. 42, pp , 1994b. J.W. Bandler, R.M. Biernaki, S.H. Chen, P.A. Grobelny, C. Moskowitz, and S.H. Talisa, Eletromagneti design o high-temperature superonduting ilters, IEEE MTT-S Int. Mirowave Symp. Dig. (San Diego, CA), pp , J.W. Bandler, R.M. Biernaki, S.H. Chen, R.H. Hemmers and K. Madsen, Eletromagneti optimization eploiting aggressive spae mapping, IEEE Trans. Mirowave Theory Teh., vol. 43, pp , 1995b. 30