397 AN OPTIMIZATION NETWORK FOR MATRIX INVERSION Ju-Seog Jag, S~ Youg Lee, ad Sag-Yug Shi Korea Advaced Istitute of Sciece ad Techology, P.O. Box 150, Cheogryag, Seoul, Korea ABSTRACT Iverse matrix calculatio ca be cosidered as a optimizatio. We have demostrated that this problem ca be rapidly solved by highly itercoected simple euro-like aalog processors. A etwork for matrix iversio based o the cocept of Hopfield's eural etwork was desiged, ad implemeted with electroic hardware. With slight modificatios, the etwork is readily applicable to solvig a liear simultaeous equatio efficietly. Notable features of this circuit are potetial speed due to parallel processig, ad robustess agaist variatios of device parameters. INTRODUCTION Highly itercoected simple aalog processors which mmc a biological eural etwork are kow to excel at certai collective computatioal tasks. For example, Hopfield ad Tak desiged a etwork to solve the travelig salesma problem which is of the p -complete class,l ad also desiged a AID coverter of ovel architecture 2 based o the Hopfield's eural etwork model?' 4 The etwork could provide good or optimum solutios durig a elapsed time of oly a few characteristic time costats of the circuit. The essece of collective computatio is the dissipative dyamics i which iitial voltage cofiguratios of euro-like aalog processors evolve simultaeously ad rapidly to steady states that may be iterpreted as optimal solutios. Hopfield has costructed the computatioal eergy E (Liapuov fuctio), ad has show that the eergy fuctio E of his etwork decreases i time whe couplig coefficiets are symmetric. At the steady state E becomes oe of local miima. I this paper we cosider the matrix iversio as a optimizatio problem, ad apply the cocept of the Hopfield eural etwork model to this problem. CONSTRUCTION OF THE ENERGY FUNCTIONS Cosider a matrix equatio AV=I, where A is a iput X matrix, V is the ukow iverse matrix, ad I is the idetity matrix. Followig Hopfield we defie eergy fuctios E Ie' k = 1, 2,...,, E 1 = (1I2)[(~ A 1j Vj1-1)2 + (~A2) Vj1 )2 +... + (~Aj Vj1)2] )-1 j-1 E2 = (1/2)[(~A1)V)2l + (~A2)V)2-1)2 + )=1 )=1 )-1 + (~A)V}2)2] }-1 America Istitute of Physics 1988
398 E = (1/2)[(~ A1J VJ)2 + (~A2J Vj )2 +... + (~A) VJ _1)2] (1) j=l }=1 J-1 where AiJ ad ViL.are the elemets of ith row ad jth colum of matrix A ad V, respectively. whe A is a osigular matrix, the miimum value (=zero) of each eergy fuctio is uique ad is located at a poit i the correspodig hyperspace whose coordiates are { V u:, V 2k ' "', V k }, k = 1, 2, "',. At this miimum value of each eergy fuctio the values of V 11' V 12'..., V become the elemets of the iverse matrix A -1. Whe A is a sigular matrix the miimum value (i geeral, ot zero) of each eergy fuctio is ot uique ad is located o a cotour lie of the miimum value. Thus, if we costruct a model etwork i which iitial voltage cofiguratios of simple aalog processors, called euros, coverge simultaeously ad rapidly to the miimum eergy poit, we ca say the etwork have foud the optimum solutio of matrix iversio problem. The optimum solutio meas that whe A is a osigular matrix the result is the iverse matrix that we wat to kow, ad whe A is a sigular matrix the result is a solutio that is optimal i a least-square sese of Eq. (1). DESIGN OF THE NETWORK AND THE HOPFIELD MODEL Desigig the etwork for matrix iversio, we use the Hopfield model without iheret loss terms, that is, --= dt a ---Ek(V 11' V 2k'..., Vk ) av ik i,k=1,2,..., (2) where u ik is the iput voltage of ith euro i the kth etwork, Vik is its output, ad the fuctio gik is the iput-output relatioship. But the euros of this scheme operate i all the regios of gik differetly from Hopfield's oliear 2- state euros of associative memory models. 3 4 From Eq. (1) ad Eq. (2), we ca defie couplig coefficiets Tij betwee ith ad jth euros ad rewrite Eq. (2) as --= dt - ~ TiJ V)k + Aki, j=l TiJ = ~ AliAIJ = Tji ' 1=1 It may be oted that T i is idepedet of k ad oly oe set of hardware is eeded for all k. The implemeted etwork is show i Fig. 1. The same set of hardware with bias levels, ~ A Ji h), ca be used to solve a liear simultaeous )=1 (3)
399 equatio represeted by Ax=b for a give vector b. INPUT OUTPUT Fig. 1. Implemeted etwork for matrix iversio with exterally cotrollable couplig coefficiets. Noliearity betwee the iput ad the output of euros is assumed to be distributed i the adder ad the itegrator. The applicatio of the gradiet Hopfield model to this problem gives the result that is similar to the steepest descet method.s But the oliearity betwee the iput ad the output of euros is itroduced. Its effect to the computatioal capability will be cosidered ext. CHARACTERISTICS OF THE NETWORK For a simple case of 3 x3 iput matrices the etwork is implemeted with electroic hardware ad its dyamic behavior is simulated by itegratio of the Eq. (3). For osigular iput matrices, exact realizatio of Tij coectio ad bias Ali is a importat factor for calculatio accuracy, but the iitial coditio ad other device parameters such as steepess, shape ad uiformity of gil are ot. Eve a complex gik fuctio show i Fig. 2 ca ot affect the computatioal capability. Covergece time of the output state is determied by the characteristic time costat of the circuit. A example of experimetal results is show i Fig. 3. For sigular iput matrices, the coverged output voltage cofiguratio of the etwork is depedet upo the iitial state ad the shape of gil'
400,...- Vm-t- --::==----r A ik > 1 = 1 < 1 Vm Ui\< Fig. 2. gile fuctios used i computer simulatios where Aile is the steepess of sigmoid fuctio tah (Aile uile)' iput matrix [ 12 I] A = -I r 1 1 0-1 (cf) 0.5 -I A-I = [ 0 1 0.5 -I -o.~] -1.5 output matrix 0.50-0.98-0.49J [ V = 0.02 0.99 1.00 0.53-0.98-1.50 o 0.5 Fig. 3. A example of experimetal results
401 COMPLEXITY ANALYSIS By coutig operatios we compare the eural et approach with other wellkow methods such as Triagular-decompositio ad Gauss-Jorda elimiatio. 6 (1) Triagular-decompositio or Gauss-Jorda elimiatio method takes 0 (8 3 /3) multiqlicatios/divisios ad additios for large X matrix iversio, ad o (2 /3) multiplicatios/divisios ad additios for solvig the liear simultaeous equatio Ax=b. (2) The eural et approach takes the umber of operatios required to calculate Tij (othig but matrix-matrix multiplicatio), that is, 0 ( 3 /2) multiplicatios ad additios for both matrix iversio ad solvig the liear simultaeous equatio. Ad the time required for output stablizatio is about a few times the characteristic time costat of the etwork. The calculatio of couplig coefficiets ca be directly executed without multiple iteratios by a specially desiged optical matrix-matrix multiplier,' while the calculatio of bias values i solvig a liear simultaeous equatio ca be doe by a optical vector-matrix multiplier. 8 Thus, this approach has a defiite advatage i potetial calculatio speed due to global itercoectio of simple parallel aalog processors, though its calculatio accuracy may be limited by the ature of aalog computatio. A large umber of cotrollable Tij itercoectios may be easily realized with optoelectroic devices. 9 CONCLUSIONS We have desiged ad implemeted a matrix iversio etwork based o the cocept of the Hopfield's eural etwork model. 1bis etwork is composed of highly itercoected simple euro-like aalog processors which process the iformatio i parallel. The effect of sigmoid or complex oliearities o the computatioal capability is uimportat i this problem. Steep sigmoid fuctios reduce oly the covergece time of the etwork. Whe a osigular matrix is give as a iput, the etwork coverges spotaeously ad rapidly to the correct iverse matrix regardless of iitial coditios. Whe a sigular matrix is give as a iput, the etwork gives a stable optimum solutio that depeds upo iitial coditios of the etwork. REFERENCES 1. J. J. Hopfield ad D. W. Tak, BioI. Cyber. 52, 141 (1985). 2. D. W. Tak ad J. J. Hopfield, IEEE Tras. Circ. Sys. CAS-33, 533 (1986). 3. J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 79, 2554 (1982). 4. J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 81, 3088 (1984). 5. G. A. Bekey ad W. J. Karplus, Hybrid Computatio (Wiley, 1968), P. 244. 6. M. J. Maro, Numerical Aalysis: A Practical Approach (Macmilla, 1982), p. 138. 7. H. Nakao ad K. Hotate, Appl. Opt. 26, 917 (1987). 8. J. W. Goodma, A. R. Dias, ad I. M. Woody, Opt. Lett. ~ 1 (1978). 9. J. W. Goodma, F. J. Leoberg, S-Y. Kug, ad R. A. Athale, IEEE Proc. 72, 850 (1984).