Parallel Approaches for Intervals Analysis of Variable Statistics in Large and Sparse Linear Equations with RHS Ranges

Transcription

1 American Journal of Applied Science 4 (5): , 2007 ISSN Science Publication Correponding Author: Parallel Approache for Interval Analyi of Variable Statitic in Large and Spare Linear Equation with RHS Range Peerayuth Charnethikul Operation Reearch and Management Science Unit, Indutrial Engineering Department Kaetart Univerity, Bangkok 10903, Thailand Abtract: Thi tudy propoe an algorithm capable of working in parallel for olving large and pare linear equation under given right hand ide (RHS) range. A comparative tudy to the direct linear programming method i reported theoretically, computationally and dicued. Moreover, the approach can be adapted for the ytem under domain decompoition tructure leading to a better efficiency experimentally. Key word: RHS range, large and pare linear equation, parallel approache, domain decompoition INTRODUCTION Thi paper conider a ytem of linear equation, AX = b with det(a) 0 and l b u where A = [a ij ] i a n matrix, b = [b i ] i a 1 uncertain vector lying between vector l = [l j ] and u = [u j ]. The problem i to olve for the poible range of X = [x j ]. Mathematically, it can be formulated a the following linear programming (LP) problem. Min (Max) x k, k = 1,2,.,n (P1) Subject to n a ij x j b i = 0, i = 1,2,...,n x j unretricted, l j b j u j, j = 1,2,..,n The above model become more complex whe i large. Uually, thi ituation arie in approximately olving linear boundary partial differential equation. Applying finite approximation cheme normally lead to the reult of very large and pare linear equation. By repreenting uncertaintie of equation right hand ide and boundary condition a a et of tatitical confidence interval, the olution a a et of related variable range can be olved uing the above formulation. Directly, the problem can be olved a a et of independent 2n LP problem. However, thi paper will preent another approach theoretically equivalent but computationally much fater a n grow and the propoed method i uitable for ue in parallel architecture. Additionally, the approach i illutrated for further application on obtaining confidence interval of variable variance/covariance matrix for a given range of right hand ide variance/covariance. Linear programming (LP) ha been one of the major tool in olving real world reource allocation problem. It origin and early development hitorically were decribed and preented extenively by Dantzig [1]. The problem of determining the confidence interval of tatitic alo ha long been tudied by tatitician [2]. A large and pare ytem of linear equation normally arie in finite approximation cheme for olving boundary value ordinary and partial differential equation baically found in engineering analyi and computational phyic. The reulting ytem i ometime too large to be handled by the direct approach and the iterative indirect method are frequently more appropriate [3,4]. The integration of thee three apect ha not yet been tudied ytematically but it application i motivated by indutrial nature of dynamic diffuion or tranport phenomena where repetitive operation caue a chance effect leading to a tationary uncertainty of ytem input or initial/ boundary condition. For a more complex ytem where the reulting equation ize can be very large, parallel computation can be more efficient. Some early work on parallel and vector olution for large linear ytem were preented in Heller [5], Ortega [6] and Stone [7]. Ditributed-memory baed parallel algorithm on baic matrix-vector multiplication epecially in cae of large blockditributed matrice were initially tudied in Dekel et al. [8]. Thi reearch area ha been received much attention and everal new approache on different Peerayuth Charnethikul, Operation Reearch and Management Science Unit, Indutrial Engineering Department, Kaetart Univerity, Bangkok 10903, Thailand 300

2 application baed oew oftware and hardware advancement have been propoed a found in Ben- Aher and Haber [9], Catalyurek and Aykanat [10], Hendrickon et al. [11], Ogielki and Aiello [12], Romero and Zapata [13] and Ujaldon et al. [14]. Extenively, thee baic parallel operation play a major role on further development of parallel iterative algorithm for large and pare linear ytem a illutrated in Baerman [15] and Ortigoa et al. [16]. For a much larger cale ytem uch a time varying three dimenional partial differential equation a found in the field of olid mechanic and diffuion procee, domain decompoition [17,18] have been a major upport for development of parallel and ditributed computing. Their main idea i to partition the original domain to everal maller domain reulting in a et of much le dependent maller ytem of linear equation that can be handled by each proceor in parallel. Then, their olution are adjuted and integrated with a et of coupling equation under manageable ize related to common variable among ub-problem. A well illutrated example can be found in Saad [4]. Recently, with the advancement of parallel hardware, parallel and ditributed algorithm for the direct Gau elimination approach have been propoed a other alternative expected theoretically that their efficiency hould be better than the iterative algorithm a preented in Balaubramanya et al. [19], Chandra and Siva Ram Murthy [20], Gallivan et al. [21] and Tufo and Ficher [22]. A pecial cae of application on both parallel matrix-vector multiplication and parallelditributing for olution of large and pare linear ytem i due to olving the teady tate dicrete and continuou Markov chain model a preented recently in Benzi and Tuma [23]. It i well known that olving an LP uing the claical implex method i an exponentially bounded [24] algorithm but it average performance wa derived by Borgwardt [25] reulting the complexity of O(m) iteration with at mot O(m 2 ) operation per iteration where m i the number of contraint in the model. In thi tudy a hown in problem P1, regardle of the bounded variable contraint, m=n. Therefore, olving the problem directly i exponentially bounded in the wort cae and polynomial bounded a O(n 4 ) on average. In thi work, an alternative algorithm will be propoed. It main computation complexity deal with computing A -1 leading to a better bound of O(n 3 ). To deign baically a parallel cheme for the algorithm, mot work reviewed in the previou paragraph can be adapted and applied. An example in cae of domain decompoition will be preliminary tudied. In ummary, there ha been a tremendou amount of work on olving large and pare linear equation in parallel due to it application and computer hardware advancement. Thee work can be ued a baic tool to upport the purpoe of olving problem P1 by the upcoming propoed algorithm. In the next ection, a et of parallel approache will be propoed and verified theoretically and experimentally. MATERIALS AND METHODS Let b i = b i l i. Therefore, 0 b i u i l i (u i ) for all i = 1, 2,..,n and n a ij x j b i = l i, i = 1,2,...,n Since det(a) 0, let A -1 = [a ij ], then x i - a ij b j = a ij l i, i = 1,2,...,n and x k = a kj b j + a kj l j, k = 1,2,...,n (1) In order to olve the correponding LP problem, the following theorem can be etablihed. Theorem: The olution vector X for the problem (P1) i optimal if and only if (1) i atified and: for each k uch that x k i maximized, for all j, b j = u j if a kj > 0, otherwie, b j = 0; for each k uch that x k i minimized, for all j, b j = u j if a kj < 0, otherwie, b j = 0; Proof: By conidering equation (1), ubtitute b j by u j when a kl > 0 and by 0 otherwie. The reult i x k * = a kj u j + a kj l j (2) (a kj >0) Comparing to equation (1), x k * x k ince b j u j for all j. Similarly, in cae of the ubtitution when a kj < 0, x k * x k. Thu, the forward part of the theorem hold. To prove the backward tatement, uppoe that the olution obtained from equation (2) i not optimal and there exit another olution for b j = c j for all j that maximize x k. The only alternative that can produce x k larger than x k * i to aign at leat one c j > u j. Thi i infeaible ince b j u j. Therefore, the revere condition hold in cae of the maximization objective and again, imilarly for the minimization cae. 301

3 Additionally, it i obviou to conclude that the above procedure i a polynomial time algorithm with the complexity of O(n 3 ) becaue the computation of A -1 i the method bottleneck. In ummary, a general method for olving problem P1 i to compute A -1 and ue ubtitute b j in equation (1) according to the theorem. However, when A i very large and pare, in cae of bandwidth matrix normally found in finite approximation for olving boundary value ordinary and partial differential equation, A -1 i fully dene. For an example, a partial differential equation in two dimenion with confidence interval of boundary value i approximated by ubdividing 1000 dicrete point in each domain leading to the ytem of linear equation with bandwidth matrix A and one million variable/equation (n = = 1,000,000). Solving for all poible variable range require a high performance computing hardware facility with an appropriate parallel algorithm. One obviou direct approach i to ditribute 2n independent LP problem of P1 among parallel proceor in order to minimize the maximum allocated proceing time. Neverthele, a more efficient method adopted from the theorem can be modified to work in parallel by olving independently for each row element of A -1 a follow. Procedure P1: Identify k (k=1,2,..,n) Solve A T z = e k = [0,0,..,1 (k th poition),,0] T and compute u k = z j u j + z j l j (z j >0) l k = z j u j + z j l j (z j <0) End. The above algorithm i deigned to break the whole operation into n independent and almot identical job. Each job i concerned with olving for a kj, j = 1,2,..,n repreenting by the vector z and can be ditributed to each available computer and the reult of u k and l k can be integrated at the end. Therefore, the deciion problem i how to allocate n job among all available m parallel proceor in order to minimize the total makepan. The difference from the original cheduling reearch i that in thi cae, the exact proceing time of each job i not known in advance but each job ha the ame intruction and the effect reulting different proceing time i the different right hand ide and the number of nonzero z j for all j in each iteration k. Comparing with the direct approach of ditributing 2n LP problem, firtly, the complexity of P1 i polynomial bounded (O(n 3 ) for the equential proceing and O(n 3 /p) for the parallel computation with p proceor) while the complexity of LP i exponentially bounded in the wort cae and O(n 4 ) in the average cae. Application in tatitic: The propoed procedure can be applied to olve the following linear tatitical model. Conider the matrix ytem of linear equation, AX = b with random vector b repreenting by expected value vector E[b] and variance/co-variance matrix K[bb T ]. In general, obtaining the exact value of E[b] and K[bb T ] may not be poible epecially in the cae that b i meaured from ome experiment or randomly generated to intimate the real ituation. If a teady tate doe exit, both tatitical parameter can be tatitically etimated a confidence interval (Chapter 7 and 10 of Lindgren [2] ). Therefore, the deciion problem i to determine the confidence interval of tatitical parameter of X (E[X], K[XX T ]) for a given confidence interval E[b], K[bb T ]. The problem of determining the confidence interval of E[X] can be olved directly uing the method decribed in the previou ection. However, the problem concerning K[XX T ] i not obviouly linked. Next, conider the following algebraic analyi. Firt, multiply equation i th and k th of a by n linear ytem and apply the expectation leading to the equation a follow. E[a ij a kl x j x l ] = E[b i,b k ] i, k= 1,2,...,n l=1 By the ymmetry property of covariance element, the equation can be adapted a follow. n E[a ij a kj x 2 j ]+2{ E[a ij a kl x j x l ]} = E[b i,b k ] i,=1,2,,n l=j+1 k=i,i+1,..,n Then, expand the expected value of each term in the equation and obtain the following equation with variance/co-variance a variable. n a ij a kj Var[x j ]+2{ a ij a kl K[x j, x l ]} = K[b i,b k ] l=j+1 i,=1,2,...,n k=i,i+1,..,n (3) Therefore, the above reulting equation ytem i linear and for a given range or confidence interval of K[b i,b k ] for all i and k, the propoed method in the previou ection can be ued to olve for the range of Var[x j ] and K[x j, x l ] for all j and l a well. However, in cae of large and pare linear ytem, n can be very large with the total number of equation/variable of 302

4 n(n+1)/2. The ytem of correponding linear ytem i o large that a ingle proceor computing ytem i inufficient for handling thi matter. Hence, a parallel algorithm i propoed a follow. Firt, the linear ytem concerning variance and co-variance of X can be written alternatively a the following matrix equation. AK[XX T ]A T = K[bb T ] Determining K[XX T ] can be performed by olving the above linear ytem. A imple and direct approach i to find A -1 and ubtitute the following relation. K[XX T ] = [A -1 ]K[bb T ][A -1 ] T (4) For A a a dene matrix, the above matrix olution can be operated within O(n 3 ) coniting of one n by n matrix inverion and two n by n matrix multiplication. Algebraically, each element in K[XX T ] can be obtained by the following equation. K[x i,x k ] = a ij a kl K[b j, b l ] l=1 i= 1,2,,n k= i,i+1,,n (5) Suppoe that a given confidence range K[b j, b l ] i [l ij, u ij ] and A i large and pare, a parallel computing cheme can be created and deigned uing the relationhip (5). The total number of independent computational job i N = n(n+1)/2 where each job ha the general decription derived in a imilar manner a the propoed procedure for the expected value cae a follow. Procedure P2: Step 0: Identify the deired i and k (i =1,2,..,n, k=i,i+1,i+2,..,n) Step 1: Let y = [y j ] and z = [z l ] and olve A T y = e i and A T z = e k. Step 2: Let Max = 0 and Min = 0 For j = 1:n For l = 1,:n If y j z l > 0 Max = Max + y j z l u jl Min = Min + y j z l l jl Otherwie, Max = Max + y j z l l jl Min = Min + y j z l u jl End End Step 3: K[x i,x k ] i within the range of [Min, Max] Suppoe that there are m available computer where m < N, a number of job can be aigned to each machine and the proceing time in tep 1 can be reduced when either i or k i fixed and another index i varied. For an example, when a computer i aigned a et of job with i = 1 and vary k from 1 to p < n, computing in tep 1 i performed merely in the full form in cae of i=1 and k=1. For another k > 1, the Am. J. Applied Sci., 4 (5): , proceing time in thi tep can be reduced in half ince only olving Az = e k i needed. An application in domain decompoition: In thi ection, both method developed in the previou ection are applied to the cae where A i formulated by the principle of domain decompoition. In general, the tructure of the correponding equation ytem i a follow. B i X i + F i Y = c i i=1,2,, (6) E i X i + CY = d (7) i=1 p i c i q i, r d t (8) where i the number of ub-domain, B i repreent coefficient quare matrix of ub-domain i with X i, c i, p i and q i a the aociated variable, RHS, lower and upper bound vector, repectively and E i and F i repreent coefficient matrice of interaction between ub-domain i and the common domain. The matrix C repreent the coefficient quare matrix of the common domain with Y, d, r and t a the aociated variable, RHS, lower and upper bound vector, repectively. To olve for the range of mean vector X i, for all i and Y, the following model can be derived baed on the imilar principle of (1) in a matrix notation. X i = Σ A ij c i + A i0 d, i =1,2,, (9) Y = Σ A 0j c i + A 00 d, i =1,2,, (10) where, A ij = B -1 i F i G -1 E j B -1 j, i,j = 1,2,,, i j A ii = B -1 i - B -1 i F i G -1 E i B -1 i, i =1,2,.. G = C Σ E i B -1 i F i i=1 A i0 = -B -1 i F i G -1, i =1,2,.. A 0j = -G -1 E j B -1 j, j =1,2,.. A 00 = G -1 For confidence interval of variable co-variance ub-matrix according to each domain interaction i and j, procedure P2 can be adapted in the following matrix form. K ij =Σ Σ A ik A jl B kl, i,j = 0,1,2,.., (11) k=0 l=0 where K ij repreent ub-matrix of co-variance between X i and X j when i,j = 1,2,.., and between X i(j) and Y when j(i) = 0 and B kl repreent ub-matrix of co-

5 variance between c i and c j when i,j = 1,2,.., and between c i(j) and d when j(i) = 0. To verify the correctne of equation (6) (11), conider AX = b formed by equation (6) and (7) a follow Simultaneouly, the ame et of problem i olved uing the direct LP command, Linprog[.], of oftware Matlab [26]. The required computation time are reported in the following table. The direct LP method No. of Variable (n) Average Time/Variable (ec.) 100,000 3, ,000 18,140 Suppoe that A -1 i in the following form. The reult from both table indicate that the propoed method i more efficient a the problem ize grow. All computation are performed on a microcomputer ytem with CPU peed of 2.4 Ghz and 1 Gbyte RAM. Moreover, the propoed method i extended to determine a ample of covariance element under variou n and their average computing time are analyzed uing 10 obervation for each problem ize and can be hown a follow. By uing the definition of A ij, it can be proven that the matrix property of AA -1 hold. Thu, A -1 i valid and can be ued in both equation (9) and (10) a claimed. Alo, a een in the block tructure decribed in equation (9), (10) and (11), firtly, the matrix G -1 mut be computed and then, parallel and ditributed computing can be independently allocated to each proceor for determining X i, Y and K ij for all i and j. Computational experience: In thi tudy, the heat tranfer equation ( 2 Ø/ x Ø/ y 2 = f(x, y)) approximated by 5 Point finite difference approach i ued with randomly generated range of f(x, y) repreented by l(x, y) and u(x, y), x = 0,, 2,., N 1 and y = 0,, 2,., N 2. i varied under the contraint that N 1 = N 2 = 1. Excluding boundary element, the total number of variable (n) for the correponding et of linear equation i (N 1-1)(N 2 1) in cae of the expected value. The propoed method wa coded in Matlab [26] utilizing it pecial command for pare linear equation olver. For each problem ize n, a ample of variable range containing 10 obervation i obtained and it average i computed and ummarized in the following table. The Propoed Method No. of Variable (n) Average Time/Variable (ec.) 100, , ,000 1, ,000 2, ,000 2, ,000 5, ,000 6, ,000 9, ,000 11,596 1,000,000 14, The Propoed Method No. of Variable No. of Covariance Average (n) Element Time/Element (ec.) 5,000 12,501, ,000 18,003,000 1,366 7,000 24,503,500 1,836 8,000 32,004,000 3,127 Since all tet cae in the above table require full dene variance/co-variance matrice, computation of each variable variance/co-variance range conume up to O(n 2 ) limiting the much maller upper bound of n a compared to the cae of expected value with O(n). The ize of n > 8000 i prohibited due to available memory. Neverthele, in practice, it i mot likely to find that the reulting variance/co-variance matrix can be approximated a a pare matrix. In thi circumtance, a et of uch large and pare K[bb T ] i randomly generated a banded matrice with k a the number of nonzero element in each row. Then, ten ample of each n ranging from 10,000 to 500,000 are teted utilizing MATLAB powerful command on pare matrix operation [26] baed on Procedure P2. The experimental reult are a hown below. No. of Variable (n) k Average Time/Element (ec.) 10, , , , , , ,211 50, ,588 50, ,965 50, , , , , , , , , ,287

6 The reult in the above table indicate ome poibilitie to conduct parallel computation in cae of large and pare variance/co-variance matrice. For a much larger cale problem, matrix partitioning and interpolation hould be further developed. It hould be noted that all computation have been conducted baed on the aumption that olving the correponding linear ytem can be performed on a ingle proceor and the propoed approach i deigned to partition the whole problem to everal independent linear ytem capable of ditributing one by one among parallel proceor. However, when the matrix A become too large to be handled by a ingle proceor, a et of cluter among proceor ha to be deigned. Each cluter conit of a et of deigned proceor capable of olving each related ytem. To upport thi concept in cae of Matlab, parallel programming i required and their baic tudy on matrix operation wa experimented in [27]. Next, a preliminary tudy on the ue of the direct approach P1 compared to uing equation (9)-(10) when the problem i in the form of equation (6)-(8) i conducted a follow. Firt, the heat equation with L- hape boundary i decompoed to 200 domain with 2500 point (variable) in each domain and 5,000 common domain point with generated data of right hand ide lower and upper limit. Then, the reulting equation in the non-decompoition mode conit of a pare 500,000 variable ytem while the domain decompoition ytem i involved with pare 2,500 2,500 B i, 2,500 5,000 F i, 5,000 2,500 E i (i =1, 2,.., 200) and 5,000 5,000 C. The experiment ha been conducted uing 4 identical proceor a pecified previouly, working in parallel with the additional main erver (2.4 GHz and 2 Gbyte RAM) working a job manager. For the direct approach, the average time out of ten ample for computing a variable interval i 2,917 ec. which i lightly le than the previou experiment cae of 500,000 variable (2940 ec.). With five parallel proceor, the expected computing time per element can be reduced by one-fifth reulting almot ten minute (2917/5). To olve the problem uing equation (9) for a pecific i, G -1 i computed in the main erver firtly. Then, computing A ij, j = 0,1,2,.., (=200 in thi cae) i ditributed equally to each proceor (=50) and A 0j i computed in parallel by the main erver. After that, the ub-vector interval for X i i computed uing equation (9) and algorithm P1 uing the main erver. At the end, 2,500 interval are obtained. In thi tudy, i =1 i conducted a an example reulting the total computation time of 57 hr. 21 min. and 6 ec. Therefore, the average time for computing a variable interval i reduced to merely one and a half minute per element. Thi baic tudy initially illutrate a further direction for continuou improvement on implementing algorithm P1 uing parallel proceing. For algorithm P2, imilar cheme can be conducted by uing equation (11). It hould be noted that the parallel trategy ued in thi cae ha not been fully developed and can be improved. For example, computing G -1 and equation (9) with i=1 have not been parallelized yet. Further exploration hould be invetigated. CONCLUSION In thi tudy, parallel approache for determining confidence interval of variable in the large and pare linear equation ytem with RHS range i propoed and preliminary teted it application poibilitie. The baic concept ha been verified and validated mathematically. Comparion with the direct linear programming approach are conducted in cae of the mean confidence interval. The tet reult illutrate that the approach greatly improve outcome efficiency. Moreover, the propoed approach can be adapted to work in parallel uing domain decompoition. An example i preliminary tudied and the initial reult indicate a further improvement. REFERENCES 1. Dantzig, G.B., Linear Programming and Extenion. Princeton Univerity Pre. 2. Lindgren, B.W., Statitical Theory. 3 rd Edn., Macmillan Publihing Co., Inc. 3. Aykanat, C., F. Ozguner, F. Ercal and P. Sadayappan, Iterative algorithm for olution of large pare ytem of linear equation on hypercube. IEEE Tran. Comput., 37: Heller, D., A urvey of parallel algorithm in numerical linear algebra. SIAM Rev., 20: Ortega, J., Introduction to Parallel and Vector Solution of Linear Sytem. Plenum Pre. 4. Saad, Y., Iterative Method for Spare Linear Sytem. PWS Publihing Co. 7. Stone, H.S., An efficient parallel algorithm for the olution of a tridiagonal linear ytem of equation. J. ACM, 20: Dekel, E., D. Naimi and S. Sahni, Parallel matrix and graph algorithm. SIAM J. Comput., 10: Ben-Aher, Y. and G. Haber, Efficient parallel olution of linear algebraic circuit. J. Parallel Ditrib. Comput., 64:

7 10. Catalyurek, U.V. and C. Aykanat, Hypergraph-partitioning-baed decompoition for parallel pare-matrix vector multiplication. IEEE Tran. Parallel Ditributed Sytem, 10: Hendrickon, B., R. Leland and S. Plimpton, An efficient parallel algorithm for matrix-vector multiplication. Internat. J. High Speed Comput., 7: Ogielki, A.T. and W. Aiello, Spare matrix computation on parallel proceor array, SIAM J. Scient. Comput., 14: Romero, L.F. and E.L. Zapata, Data ditribution for pare matrix vector multiplication. J. Parallel Comput., 21: Ujaldon, M.U., E.L. Zapata, S.D. Sharma and J. Saltz, Parallelization technique for pare matrix application. J. Parallel Ditributed Comput., 38: Baerman, A., Parallel pare matrix computation in iterative olver on ditributed memory machine. J. Parallel Ditrib. Comput., 45: Ortigoa, E.M., L.F. Romero and J.I. Ramo, Parallel cheduling of the PCG method for banded matrice riing from FDM/FEM. J. Parallel Ditrib. Comput., 63: Farhat, C. and P.S. Chen, Tailoring domain decompoition method for efficient parallel coare grid olution and for ytem with many right hand ide. Contem. Math., 180: Smith, B., P. Bjortad and W. Gropp, Domain Decompoition, Cambridge Univ. Pre Cambridge, MA. 19. Balaubramanya, K.N.B., Murthy, C. and Siva Ram Murthy, Gauian elimination baed algorithm for olving linear equation on meh connected proceor. IEE Proc. Comp. Digit. Tech., 143: Chandra, R., C. Siva Ram Murthy, A fater algorithm for olving linear algebraic equation on the tar graph. J. Parallel Ditrib. Comput., 63: Gallivan, K., R.J. Plemmon and A.H. Sameh, Parallel algorithm for dene linear algebra computation. SIAM Rev., 32: Tufo, H.M. and P.F. Ficher, Fat parallel direct olver for coare grid problem. J. Parallel Ditributed Comput., 61: Benzi, M. and M. Tuma, A parallel olver for large-cale Markov chain. Appl. Numer. Math., 41: Klee, V. and G. Minty, How good i the Simplex-Algorithm?, Inequalitie III, O. Shiha (Ed.), Academic Pre, New York, pp: Borgwardt, K.H., The Simplex Method: A Probabilitic Analyi. Springer-Verlag. 26. The Math Work, Inc. MATLAB manual and intruction et, Miloavljevic, I.Z. and M.A. Jabri, Experimental evaluation of automatic array alignment in parallelized Matlab, J. Parallel Ditributed Comput., 61: