Analysis of an Algorithm for Identifying Pareto Points in Multi-Dimensional Data Sets

Transcription

1 Analysis of an Algorithm for Identifying Pareto Points in Multi-Dimensional Data Sets Michael A. Yukish * and Timothy W. Simpson The Pennsylvania State University, University Park, PA, In this paper we present results from analytical and experimental investigations into the performance of divide & conquer algorithms for determining Pareto points in multidimensional data sets of size n and dimension d. The focus in this work is on the worst-case, where all points are Pareto, but extends to problem sets where only a partial subset of the points is Pareto. Analysis supported by experiment shows that the number of comparisons is bounded by two different curves, one that is O(n (log n)^(d-2)), and the other that is O(n^log 3). Which one is active depends on the relative values of n and d. Also, the number of comparisons is very sensitive to the structure of the data, varying by orders of magnitude for data sets with the same number of Pareto points. Nomenclature n = number of points in a data set d = dimension of the data set TZ,, = Table of n records, each record having d attributes tz,, = A record with d attributes ti, z i, = The ith attribute in a record DC = Divide & Conquer algorithm pbf[n,d] = estimator for number of comparisons in DC algorithm, n points and dimension d mbf[n,d] = estimator for number of comparisons in marriage step of DC algorithm I. Introduction N this paper we present results from analytical and experimental investigations into the performance of divide & I conquer (DC) algorithms for determining Pareto points in multi-dimensional data sets of size n and dimension d. The focus in this work is on the worst-case, namely, where all points are Pareto, but extends to problem sets where only a partial subset of the points is Pareto. The motivation for this work stems from ongoing research to improve the computational design process for large, complex systems such as aircraft, land vehicles, or spacecraft. In particular, this work supports a paradigm shift from computational design optimization to design exploration. Whereas past research has focused on specifying constraints and objectives a priori for a system and generating a best design or a set of best designs in accordance with those objectives, 1-6 the new paradigm is to generate thousands or millions of candidate designs across the trade space, store them in a database, and then allow a user to express their constraints and objectives a posteriori and view the best designs from the set of candidate designs. 7-9 The effort in design exploration is supported by ongoing research in the areas of design automation, metamodeling and response surface techniques, and by research in methods to explore design trade spaces visually. As an example, Figure 1 shows visualization of the Pareto frontier in the Advanced Trade Space Visualization (ATSV) software developed by a team of researchers that includes the authors. 10 In the figure, the nondominated points are indicated by black crosses. As users move the slider bar at the bottom of each screen, they adjust the constraints, and a new set of points is feasible. The interface must recalculate and display the Pareto frontier in real time with large numbers of points. This turns the problem of identifying the Pareto frontier from a fixed set of points into a computing problem analogous to the sorting problem for 1D data, or identifying the convex hull for higher dimensional data. 11 The need to update in real time drives the goal of efficient computation. * Head, Product & Process Design Dept., Applied Research Laboratory, may106@psu.edu. Associate Professor of Mechanical Engineering and Industrial Engineering, Member AIAA, tws8@psu.edu. 1

2 User s constraint applied to data Figure 1. Visualizing the Pareto points (black crosses) as user dynamically varies the data set The literature on algorithms for identifying the Pareto set within a multi-dimensional data set is surprisingly sparse, consisting mainly of three papers by Kung 12 and Bentley. 13,14 The paper by Kung and the first paper by Bentley develop the divide & conquer (DC) algorithm. While the two papers provide an order of complexity for the algorithm, they do not develop actual estimates for the number of comparisons. The second paper by Bentley develops an expected linear run time algorithm, but this algorithm is predicated on there being no correlation between each point s values in the different dimensions. This assumption results in relatively few numbers of Pareto points as a percentage of the total data set. The assumption breaks down for data with correlation between dimensions and also for uncorrelated data of high dimensionality. Our experience with complex systems design problems shows that the uncorrelated assumption does not hold in general. There has also been recent work in this area in the constraint programming community, 15,16 but the common view is that of Ehrgott and Gandibleux 17 who in their annotated bibliography of multiobjective combinatorial optimization consider the problem efficiently solvable and not of mathematical interest. Whereas similar problems such as sorting continue to draw attention, algorithms to identity Pareto points from data sets have not. A probable reason for this view is that there has not been a requirement for rapid, continual updating of Pareto sets, as is needed in visualization tools that support design space exploration. Apart from visualization, as design exploration becomes more prevalent, large data sets will be used, and so algorithmic efficiency for computing the Pareto points will become increasingly important. There has been recent activity in the evolutionary computing literature as researchers have developed a need to efficiently identify nondominated points In these works O(n 2 ) algorithms have been developed to identify the nondominated points. A recent paper in the evolutionary computing literature 24 has identified the algorithms of Kung and Bentley as applicable to the evolutionary computing problem and conducted empirical tests to demonstrate their efficacy. The DC algorithm was first presented by Kung 12 and then further developed and analyzed by Bentley, 13,14 with the focus being on randomized data sets with relatively few numbers of Pareto points compared to the total number d 1 of points in the data set. The number of comparisons was shown by Kung to vary in proportion to On ( log n), while Bentley showed that if one could execute sorting procedures in linear time (after the initial On ( log n) sorting d 2 of the data), that the asymptotic efficiency is On ( log n). Analysis supported by experiment shows that the number of comparisons is in fact bounded by two different d 2 curves, one that is On ( log n), and the other that is On ( ). Which one is active depends on the relative values of d 2 n and d. The On ( ) bound is active for data sets with smaller n and larger d, whereas the On ( log n) bound holds otherwise. The On ( ) bound can be viewed as the bound for the algorithm operating on data sets of infinite dimension. Also, the number of comparisons is very sensitive to the structure of the data, varying over orders of magnitude for data sets with the same number of Pareto points. Experiments have shown that for data of a particular structure, by simply reordering the columns of data differences in number of comparisons of a factor of 1000 have been seen. This sensitivity to the ordering of columns suggests a possible avenue for creating an adaptive algorithm that is more efficient than one that simply takes the columns in their original order. Such an algorithm has been developed 25 and will be introduced in future papers. 2

3 We also present estimating relationships of the expected number of comparisons for the DC algorithm as a function of n and d, along with experimental results that validate these relationships. In particular, analytical estimates are developed for the number of comparisons, the number of function calls in the recursion, and the amount of data passed into each function call. The remainder of this paper is as follows. Section II explains the DC algorithm as repeated operations against a table of data in order to simplify the worst-case analysis. Section III analyzes the infinite dimensional case and shows that it solves in On ( ). Section IV examines the finite dimension case and shows that the infinite dimension result is a limiting case to the finite dimension problem. Section V validates the analytical results with experimental data and demonstrates the sensitivity of the algorithm s number of comparisons to the structure of the data. Section VI extends to the number of comparisons analysis to data sets with only a subset of the points Pareto. Section VII provides a summary and conclusions. II. Description of DC Algorithm Bentley describes the DC algorithm using a geometric analogy. 13 In this section, rather than treat the data as a multi-dimensional space, consider the data as a table T with d columns and n rows. There exists a total ordering within each column, although there may be no possible comparisons between columns, e.g., one column may be based on a lexicographic ordering of names while another may be a column of integers in numeric order. In the first step of the DC algorithm, the points are sorted based on their first column, split into two halves based on that column, and the nondominated points in each half are identified. As an example, consider the data set of 18 points shown in Figure 2. The data is split on the first column, and points 1-9 and are treated as separate sets Figure 2. Data sorted on first column and split in half Once the nondominated points in each subset are identified, the two sets must be married in a marriage step. The marriage process removes elements from the second set that are dominated by elements in the first. Let the points in the set of points 1-9 that are nondominated be collectively labeled as A, and the points from the set be collectively labeled as B (see Figure 3). Each of the A points are nondominated within their set, and they cannot be dominated by a B point, since the value for the first column for every point in B is greater than the value of the first column for every point in A. Each of the B points are nondominated within their set, but can be dominated by points in A. The object of the marriage step is to remove the points in B that are dominated by points in A. A A A A A A A A A B B B B B B B B B Figure 3. After culling the two subsets In Figure 3, assuming that for each set of 9 points only 8 of them are nondominated within their set, the total number of points remaining is now 16; each set having lost one point. In the next step, the points are resorted, this time on the second column and again broken into two sets (see Figure 4). Each set, on the left and the right, contains A and B points. Label the left set as L and the right set as R. A B A A B B B A A A A B A B B B Figure 4. Points resorted on the second dimension The algorithm then compares the B s in set L with the A s in set L, and any B s that are dominated are removed, similarly for R. In this case, assume that all points remaining are nondominated, and therefore no points are removed from either side. At this point, what is known is that all of the B s in set L are nondominated by any A s, left or right. Also, no A s in set R dominate any B s, left or right. What is not known is whether any A points on the left dominate any of the B points on the right. Mark the left B s and the right A s in grey (see Figure 5). In the next step, since left B s and right A s cannot dominate or be dominated, they do not be to be considered in the algorithm and can be dropped prior to the next step. 3

4 A B A A B B B A A A A B A B B B A A A A B B B B Figure 5. Left B s and right A s marked and then removed Again resort the data, now on the third column, and repeat. Keep doing so until no points remain. Ready for resort A A A A B B B B Sorted on next index A B B A B A B A After checking for dominance, drop B s from left and A s from right A B B A B A B A A A B B Ready for next resort Figure 6. Cycle of resort, check for dominance left and right, drop left B s and right A s There are three possible situations one ends up with in the resort. The first case, shown in Figure 2 - Figure 6, is where the A s and B s are distributed on both sides of the split. The second case is where the left side has all B s and the right side has all A s after the resort. This indicates that all points are nondominated with respect to each other, and the algorithm can return immediately. The final case is where, after the resort, the left side contains all A s and the right side contains all B s. In this situation, the algorithm will immediately resort the data based on the next chosen column and continue. The critical point to note is that as long as the nondominance of any two points with respect to each other can be established in a finite number of steps of the algorithm, the algorithm will complete even for a set of points of infinite number of columns/dimensions. The result stems from the fact that the left B points and right A points are discarded each time the algorithm steps to resort on the next column. Given that this is true, it is possible to determine the worst-case number of comparisons for a set of points of infinite dimension with all points nondominated. From there, the finite dimensional problem can then be described and solved. In developing the estimate of number of comparisons, we make the assumption that at each resorting on columns, the resort is balanced, i.e., after the resort the L set will have the same number of A and B points, and similar for the R set. Experiments show that the estimators based on the assumption are accurate. 25 III. Infinite Dimension Comparisons Assuming the problem is of infinite dimension but that the nondominance of any two points with respect to each other can be established in a finite number of steps of the algorithm, worst-case number of comparisons of the algorithm can be determined. We will assume that the distribution of A and B points in the L and R sets when each step in dimension is made is balanced, i.e., at each resort in dimension, A and B points are equally distributed in sets L and R (see Figure 6). Although the algorithm developed here is a divide & conquer algorithm, conceptually it is easier to consider it as an initial divide followed by repeated marrying of fewer but larger sets. Define mbi[n] (mbi is short for Marry Balanced Infinite) as a function that determines the number of comparisons required to marry n points together (i.e., marry two sets of n/2 points). Define a function pbi[n] (for Pareto Balanced Infinite) to determine the number of comparisons required to verify that a set of n points is nondominated. Then there are two recursive relationships, one for each of the two functions. For the marrying of a set of balanced points of infinite dimension, the recursion is n mbi[ n] = 3mbi 2. This recursion derives from the fact that the marriage step initially splits the problem of size n into two subproblems of size n/2, and then resorts on the next column and marries the remainder. Since the problem is balanced, half of the points are dropped prior to the final call so that three calls are made on data sets of size n/2. When determining the Pareto points for the balanced case of infinite dimension the recursion is 4

5 n pbi[ n] = 2pbi + mbi[ n] 2. For the DC algorithm, two recursive calls to the DC algorithm, each with data sets of size n/2, are followed by a call to the marrying algorithm with a data set of size n. The base case for inductive purposes is the case n=2, where mbi[2]=1 and pbi[2]=1. Assuming that n is a power of 2, the recursions can be resolved directly. For the marry portion, the recursion resolves to 1 mbi[ n] = n 3 and pbi[n] resolves to pbi[ n] = n n. For the infinite dimension case, the number of comparisons is therefore bounded by On ( ). IV. Finite Dimension Comparisons For the finite dimension case, the dimension of the problem considered becomes important. The recursive relationship for the marriage step in the finite balanced case, referred to as mbf[ ], is mbf[ n, d] = 2 mbf[ n/ 2, d] + mbf[ n/ 2, d 1] with the base cases being mbf[2, d] = 1 mbf[ n, 2] = n 1. The recursions for the Pareto algorithm for the finite balanced case, referred to as pbf[ ], is pbf[ nd, ] = 2 pbf[ n/ 2, d] + mbf[ nd, 1] with the base cases being pbf[2, d] = 1 pbf[ n, 2] = n 1. The resulting expressions for mbf[n,d] and pbf[n,d] are more complex than their infinite dimension counterparts. The cases for dimension 3 through 8 are shown in Figure 14 and Figure 16 in the Appendix. Note that the number of comparisons varies by O(nlog d-2 n) as expected. What is interesting is the plot of comparisons versus dimension shown in Figure 7. One would expect that, fixing the number of points in a data set, the number of comparisons would exponentially increase with dimension. Instead, for a fixed number of Pareto points, the curve increases with increasing dimension of the problem until reaching a bounding level. This bounding level is in fact the infinite dimension value pbi[n] (see Figure 8). Number of Comparisons Dime Figure 7. Number of comparisons as a function of dimension for fixed number of points in problem number of Comparisons dimension Figure 8. Plot of number of comparisons versus dimension for 10,000 point problem, solid line is pbf[n,d], dotted line is pbi[n] 5

6 V. Comparison with Experimental Data Figure 9 shows results of both experiments and analyses in estimating the number of comparisons for problems with all points nondominated, varying problem size, and varying dimension. For the analytical results the balanced case function pbf[n,d] was used. Experimental results were derived by using the random dominated point (RDP) algorithm 25 to generate data of dimension ranging from 2 to 60. The RDP algorithm generates a random lattice with an expected number of Pareto points for a data set of size n and dimension d. The RDP algorithm has been compared against several methods for generating random test data and is one of the fastest algorithms that allows the user to specify the expected number of Pareto points in the data set. 25 The results show that the experimental values approach the analytical values as the dimension increases. Number of Comparisons Dim 250 experimental 500 experimental 1000 experimental 1500 experimental 250 analytical 500 analytical 1000 analytical 1500 analytical Figure 9. Experimental and analytical results up to dimension of 60 One can estimate a scaling coefficient α to multiply with pbf[ ] function to improve the fit for each dimension. The Appendix lists the results of the curve fits. Figure 10 shows the fit between the pbf[ ] function scaled by α and the data used to fit the α coefficient. Dimensions 3 through 18 are shown in Figure 10. Comps d= Comps d= Comps d= Comps d= Comps d= Comps d= Figure 10. Data versus curve fits for the α pbf[ ] function with varying dimension Finally, Figure 11 basically repeats Figure 9 but with the pdf[n,d] value multiplied by the mean value for the scaling coefficient of α = 0.78 from Table 1 in the Appendix, resulting in better correspondence with the experimental data. This estimate can be used as an approximate value of the number of comparisons required for random data with known dimension and all points nondominated. For a more accurate fit, the mean value for the scaling coefficients could be replaced with the dimension-specific values, also in Table 1. 6

7 Number of Comparisons Dim 250 experimental 500 experimental 1000 experimental 1500 experimental 2500 experimental 250 analytical 500 analytical 1000 analytical 1500 analytical 2500 analytical Figure 11. Comparison of empirical data with the analytical estimate gained by multiplying pbf[n,d] by mean scaling value of α=0.78 The comparison of experimental with analytical results validates the proposition that the number of comparisons of the algorithm is bounded by two distinct curves that are O(n ) and O(nlog d-2 n). The algorithm proves sensitive to structure in the data. Interestingly, it is possible to get vastly different performance out of two data sets that both are constituted of only nondominated points. Consider the following data set of 6 dimensions: {{ 1,1,1,1,1,1000 },{ 2, 2, 2, 2, 2, 999 },,{ 1000,1,1,1,1,1} } The number of comparisons needed to execute the algorithm is 11,207. Conversely, look at this data set, identical to the previous but for permuting the order of columns: {{ 1000,1,1,1,1,1 },{ 999,2,2,2,2,2 },,{ 1000,1,1,1,1,1} } The number of comparisons for this data set is only 536, which is less than the original unpermuted data set by a factor of 20. The reason for the improved performance against the data set lies in the marriage step of the algorithm. For the second data set, upon entry into the marriage step, the L set will consist solely of B points, and the R set will consist solely of A points. This stops the recursion, and the algorithm returns immediately. For the first data set, during the marriage steps, the L and R sets will be occupied with A and B points, respectively, until the 5 th column is used for sorting. This results in the great increase in the number of comparisons. In order to overcome this lack of determinism and disparity in number of comparisons, the algorithm can be modified to randomly choose the next column of data upon which to split. The potential exists for prior analysis of the data to take advantage of structure and order the columns to minimize the number of comparisons. VI. Estimating Number of Comparisons as a Function of Number of Pareto Points So far the assumption has been that all points are Pareto. Experiments show that the relationship between the number of comparisons and the percentage of points that are nondominated is approximately linear. Figure 12 shows this approximately linear relationship, with individual plots for problems of dimensions 2 through 6. 7

8 Comparisons Number of Nondominated Points Figure 12. For problem size of 1000 points, the number of comparisons as a function of dimension and of the number of nondominated points An approximation for the number of comparisons needed in the DC algorithm is p n p pbf[ pnd,, ] = αd pbf[ nd, ] + spp[ nd, ] n n where p is the number of Pareto points, n is the total number of points, spp[n,d] is a function to estimate the number of comparisons in a data set with only one Pareto point, and α d is a parameter to fit the analytical estimates to experimental results (see Table 1 in Appendix). The function spp[n,d] is sensitive to the structure of the data. One possible structure for the data is for there to be a single Pareto point, and for the remaining data to be all Pareto if the single dominant point was removed. A simple recursion can be developed for this case: spp1[ nd, ] = spp1[ n/2, d] + pbf[ n/2, d] + n. In this case, the spp1[n,d] function is at least as large as pbf[n/2,d], which is clearly not supported by the experimental results. Another possible recursion is to assume that each possible subset of the data has a single Pareto point, which leads to the recursion spp 2[ nd, ] = 2spp 2[ n/ 2, d] + 1. This leads to an estimate of spp2[n,d] = n. The conditions supporting the use of spp2[n,d] are much more likely to be encountered as opposed to the conditions supporting spp1[n,d], as the probability that a random data set has a single Pareto point is 1 1 d, N 6 which for a 3D data set with 1000 points is, for example, 10. The probability that a data set that is otherwise inversely correlated would have a single Pareto point is small. Adopting the form of spp2[n,d] as spp[n,d] leads to p pbf[ pnd,, ] = α d pbf[ nd, ] + ( n p) n. (1) One implication of Eq. (1) is on algorithms to do Pareto Sorting, where a Pareto algorithm repeatedly identifies the first, second, third, etc. Pareto frontier of a data set. Researchers within the evolutionary computing domain have been concerned that repeatedly running the DC algorithm to identify the layers of Pareto frontiers could result in run 2 d 2 2 times that scaled by On ( log n), and have developed more complicated algorithms with On ( ) storage to alleviate this concern. Eq. (1) suggests this not to be the case, and that repeatedly running the DC algorithm would be efficient. VII. Summary and Conclusions As stated in the introduction, the number of comparisons of the DC algorithm is bounded by both O(n ) and by O(nlog d-2 n), with the former bound active with smaller data sets and higher dimensionality. Estimators of the number of comparisons of the DC algorithm were developed and validated against experimental data. The sensitivity of the DC algorithm to structure in the data was demonstrated through a simple contrived case. The sensitivity to structure immediately suggests potential improvements to the algorithm that might scan the data first to determine the best ordering of the data s columns prior to executing the core of the DC algorithm. This is an open area of research. 8 d=2 d=3 d=4 d=5 d=6

9 The authors have developed a hybrid Pareto algorithm 25 that is a mix of the DC algorithm and a simpler algorithm that is nonrecursive but O(n 2 ). The hybrid algorithm operates recursively as per the DC algorithm until the subdivided data sets reach a critical size, and then switches to the nonrecursive algorithm. Reductions in number of comparisons of 60% over the DC algorithm are typical. The algorithm will be presented in future papers. Appendix Table 1 lists the scaling factors to fit the pbf[ ] function to the experimental data. The scaling varies as a function of the dimension of the data. Table 1. Scaling factors for the pbf[ ] function to fit to data, Dimension Scaling Factor Dimension Scaling Factor Mean The following figures show the expressions for the analytical estimates for number of comparisons in the DC algorithm, pbf[ ], and the number of comparisons in the marriage step, mbf[ ]. The Mathematica code for generating the analytical estimates of the number of comparisons during the marriage step is in Figure 13. The code generates estimates up to d= 20, but can be modified to generate estimates for any range. The code is exercised for d=5. In[9]:= mbf@n_, 2D := n 1; mbf@2, d_d := 1; i 1 Log@2,nD i mbfa 2i,d 1E yy TableAmbf@n_, dd := EvaluateA n i E, 8 d, 3, 20<E; k i=2 k {{ mbf@n, 5D Out[12]= n i 1 k Log@2D3 54 n Log@2D nLog@2D 2 Log@nD 6nLog@2D Log@nD 2 + nlog@nd 3 y 48 n Log@2D 3 { Figure 13. Mathematica code for generating analytical estimate of number of comparisons for the Marry algorithm Note in the result for mbf[n,5] in Figure 13 the expression Log[2] appearing in the numerator and the denominator. This is because Mathematica converts all Base 2 logarithms such as log 2 n into the equivalent form ln n/ln 2. In Mathematica Log[n] indicates the natural logarithm of n. To simplify presentation of the results below, Mathematica s output has been modified so that log[n] (note the lower case l in the function) indicates the logarithm Base 2 of n. Figure 14 shows the estimates for the Marry algorithm for dimensions 2 through 8. All logarithms are in Base 2. 9

10 Dimension n 3 1 n n log@nd n n log@nd n log@nd n n log@nd 1 8 n log@nd nlog@nd n nlog@nd n log@nd n log@nd n log@nd n nlog@nD 47 n log@nd n 256 log@nd3 1 n 256 log@nd4 + nlog@nd n nlog@nD n 720 log@nd2 43nlog@nD3 + 59nlog@nD4 7nlog@nD5 + nlog@nd Figure 14. Expresssions for mbf[ ] Figure 15 shows the code for generating the estimates of number of comparisons for the DC algorithm. Figure 16 shows pbf [ ] function for dimensions 2 through 8. In[51]:= pbf@n_, 2D := n 1; pbf@2, d_d := 1; Log@2,nD i 1 i TableApbf@n_, dd := EvaluateA n 2 + mbf@2 i,d 1D y y k 2 i E, 8 d, 3, 20<E; { k i=2 { Figure 15. Mathematica code for generating analytical estimate of number of comparisons for the DC algorithm Dimension pbf@n,dd n 3 1 n + n log@nd n 1 4 n log@nd n log@nd2 5 1 n nlog@nd 1 8 n log@nd n log@nd n 9 nlog@nd nlog@nd n log@nd nlog@nd4 7 1 n n log@nd n log@nd nlog@nd n log@nd4 + nlog@nd n n log@nd2 nlog@nd n log@nd3 + 41nlog@nD4 nlog@nd5 + nlog@nd Figure 16. Expressions for pbf[ ] References 1 Hazelrigg, G. A., Systems Engineering : an Approach to Information-Based Design, Prentice Hall, Upper Saddle River, NJ, Allen, B., "On the Aggregation of Preferences in Engineering Design," ASME Design Engineering Technical Conferences - Design Automation Conference (Diaz, A., ed.), Pittsburgh, PA, ASME, Scott, M. J. and Antonsson, E. K., "Arrow's Theorem and Engineering Design Decision Making," Research in Engineering Design, Vol. 11, No. 4, 1999, pp Shah, J. J. and Wright, P. K., "Developing Theoretical Foundations for DfM," ASME Design Engineering Technical Conferences - Design for Manufacturing, Baltimore, MD, ASME, Tang, X. and Krishnamurty, S., "On Decision Model Development in Engineering Design," Journal of Engineering Valuation & Cost Analysis, Special Edition on Decision-Based Design, Vol. 3(2/3), 2000, pp Thurston, D. L., "Real and Misconceived Limitations to Decision Based Design with Utility Analysis," ASME Journal of Mechanical Design, Vol. 123, No. 2, 2001, pp Stump, G., Yukish, M., Simpson, T. and Bennett, L., "Multidimensional Visualization and Its Application to a Design by Shopping Paradigm," 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Washington, DC, AIAA, Yukish, M. and Harris, E., "Formulations To Support the Conceptual Design of Satellites," World Space Congress 2002, Houston, TX, AIAA,

11 9 Balling, R., "Design by Shopping: A New Paradigm?," Proceedings of the Third World Congress of Structural and Multidisciplinary Optimization (WCSMO-3) (Bloebaum, C. L., Lewis, K. E. and Mayne, R. W., eds.), Buffalo, NY, University at Buffalo, Vol. 1, 1999, pp Stump, G., Yukish, M., Simpson, T. and O'Hara, J., "Trade Space Exploration of Satellite Datasets Using a Design by Shopping Paradigm," IEEE Aerospace Conference, New York, IEEE, O'Rourke, J., Computational Geometry in C, Cambridge University Press, Cambridge, Kung, H. T., Luccio, F. and Preparata, F. P., "On Finding the Maxima of a Set of Vectors," Journal for the Association of Computing Machinery, Vol. 22, No. 4, 1975, pp Bentley, J., "Multidimensional Divide-and-Conquer," Communications of the ACM, Vol. 23, No. 4, 1980, pp Bentley, J., Clarkson, K. and Levine, D., "Fast linear expected-time algorithms for computing maxima and convex hulls," Annual ACM-SIAM Symposium on Discrete Algorithm, 1990, pp Gavanelli, M., "An Implementation of Pareto Optimality in CLP(FD)," CP-AI-OR-International Workshop on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems (Jussein, N. and Laburthe, F., eds.), Le Croisic, France, Ecole des Mines de Nante, 2002, pp Gavanelli, M., "Partially Ordered Constraint Optimization Problems," Principles and Practice of Constraint Programming, 7th International Conference-CP2001 (Walsh, T., ed.), Paphos, Cyprus, Springer Verlag, Vol. 2239, Ehrgott, M. and Gandibleux, X., "An Annotated Bibliography of Multiobjective Combinatorial Optimization," NR 62/2000, Universitat Kaiserslautern, Kaiserslautern, Germany, Deb, K., Multi-Objective Optimization using Evolutionary Algorithms, Wiley, New York, Deb, K., Pratab, S., Agarwal, S. and Meyarivan, T., "A Fast and Elitist Multiobjective Genetic Algorithm: NGSA-II," IEEE Transactions on Evolutionary Computing, Vol. 6, 2002, pp Knowles, J. D. and Corne, D. W., "Approximating the Nondominated Front Using the Pareto Archived Evolutionary Strategy," Evolutionary Computing, Vol. 8, 2000, pp Murata, T. and Ishibuchi, H., "MOGA: Multi-Objective Genetic Algorithms," IEEE Conference on Evolutionary Computation, New York, IEEE, 1995, pp Zitzler, E., Deb, K. and Thiele, L., "Comparison of Multiobjective Evolutionary Algorithms: Emprical Results," Evolutionary Computing, Vol. 8, No. 2, 2000, pp Zitzler, E. and Thiele, L., "Multiobjective Evolutionary Algorithms: A Comparative Case Study and the Strength Pareto Approach," IEEE Transactions on Evolutionary Computing, Vol. 3, 1999, pp Jensen, M. T., "Reducing the Run Time Complexity of Multiobjective EAs: The NGSA-II and Other Algorithms," IEEE Transactions on Evolutionary Computing, Vol. 7, No. 5, 2003, pp Yukish, M., "Algorithms to Identify Pareto Points in Multi-Dimensional Data Sets," Ph.D. Dissertation, Mechanical Engineering Dept., The Pennsylvania State University, State College, PA,