Author s Accepted Manuscript

Transcription

1 Author s Accepted Manuscript 3D-RadVis Antenna: Visualization and Performance Measure for Many-objective Optimization Amin Ibrahim, Shahryar Rahnamayan, Miguel Vargas Martin, Kalyanmoy Deb PII: DOI: Reference: To appear in: S (16) SWEVO314 Swarm and Evolutionary Computation Received date: 27 November 2016 Revised date: 10 July 2017 Accepted date: 22 September 2017 Cite this article as: Amin Ibrahim, Shahryar Rahnamayan, Miguel Vargas Martin and Kalyanmoy Deb, 3D-RadVis Antenna: Visualization and Performance Measure for Many-objective Optimization, Swarm and Evolutionary Computation, This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting galley proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

2 1 3D-RadVis Antenna: Visualization and Performance Measure for Many-objective Optimization Amin Ibrahim, Shahryar Rahnamayan, SMIEEE, Miguel Vargas Martin, Member, IEEE, and Kalyanmoy Deb, Fellow, IEEE Abstract So far the focus of almost all multi- or many-objective performance metrics has been the convergence and distribution of solutions in the objective space (Pareto-surface). Pareto-surface metrics such as IGD, HV, and Spread are simple and provide knowledge about the overall performance of the solution set. However, these measures do not provide any insight into the distribution or spread of a solution set with respect to each objective. Further, in many-objective optimization, visualization of true Pareto fronts or obtained non-dominated solutions is difficult. A proper visualization tool must be able to show the location, range, shape, and distribution of obtained non-dominated solutions (both Pareto-surface and objective-wise distribution). Existing commonly used visualization tools in many-objective optimization (e.g., parallel coordinates) fail to show the shape of the Pareto front or distribution of solutions along each objective. In this paper, we propose an extension of recently proposed visualization method called 3D-RadVis (we call it 3D-RadVis Antenna) to visualize the distribution of solutions along each objective. 3D-RadVis Antenna is capable of mapping M- dimensional objective space to a 3-dimensional radial coordinate plot while seeking to preserve the relative location of solutions, shape of the Pareto front, and distribution of solutions along each objective. Furthermore, 3D-RadVis Antenna can be used by decisionmakers to visually navigate large many-objective solution sets, to observe the evolutionary process, to visualize the relative location of a solution, to evaluate trade-offs among objectives, and to select preferred solutions. Along with this visualization tool, we propose two novel performance measures, named objective-wise inverse generational distance (ObjIGD) and line distribution ( ) to measure the convergence and distribution of solutions along each objective as well as the overall performance of approximate solutions. The effectiveness of the proposed methods are demonstrated on widely used many-objective benchmark problems containing a variety of Pareto fronts (linear, concave, convex, mixed, and disconnected). In addition, for a case study, we have demonstrated the capability of 3D-RadVis Antenna combined with the proposed performance measures for visual progress tracking of the NSGA-III algorithm through generations. Experimental results show that the proposed visualization method can effectively be used to compare and track the performance of many-objective algorithms. Moreover, the proposed measures can be used as reliable complementary measures along with other widely used performance measures to compare many-objective solution sets. Index Terms Visualization, performance metrics; evolutionary computation; many-objective optimization, radial coordinate mapping; 3D-RadVis; convergence; diversity; hypervolume. R I. INTRODUCTION EAL-WORLD optimization applications are ever more encompassing and increasingly complex these days. Particularly, more input data and parameters are available to capture the complexity of a problem leading to more decision variables being used to model complex situations. Additionally, optimizing a high number of objectives is involved in these situations, leading to further complexity. Many real-world applications involve a high number of objectives (typically more than three). Visualization of solutions hence becomes difficult. However, visualization is necessary as it is a proper decision making tool leading to a better understanding of algorithms used and tradeoff solutions. As the number of objectives exceeds three, the visualization of approximation sets is more challenging [1]. There exists many 2- or 3-dimensional data visualization methods that are used for many-objective optimization (MaOO). Parallel coordinates [2] and Heatmap plots [3] are two examples that can be used to visualize distribution, range, and A. Ibrahim is with the Faculty of Electrical, Computer, and Software Engineering, University of Ontario Institute of Technology, Oshawa, Canada (amin.ibrahim@uoit.ca) S. Rahnamayan is with the Faculty of Electrical, Computer, and Software Engineering, University of Ontario Institute of Technology, Oshawa, Canada (shahryar.rahnamayan@uoit.ca) M. Vargas Martin is with the Faculty of Business and Information Technology, University of Ontario Institute of Technology, Oshawa, Canada (miguel.vargasmartin@uoit.ca) K. Deb is with the Faculty of Electrical, Computer, and Computer Engineering Michigan State University, East Lansing, Michigan, USA (kdeb@egr.msu.edu)

3 trade-off among solutions of multi-dimensional objectives. Nonetheless, such methods are often difficult to interpret due to solutions being superimposed or arbitrarily ordered [4]. There are other methods, such as self-organizing maps [3] and radial coordinate visualization [5], that show the distribution and inter-relationship among objectives, yet they do not illustrate the shape and convergence trend of the solution sets. Fortunately, there are recent advances in visualization methods to cope with visualizing high-dimensional search space while attempting to preserve the distribution, shape, and dominance relationship among approximate Pareto front members. Some of these visualization methods include the extension of radial coordinate visualization and Heatmap [4, 6-8]. The other main issue is the lack of quantitative metrics capable of measuring the convergence and diversity of a solution set when the number of objectives is high. For example, the hypervolume (HV) [9] measure is widely used performance metric in MaOO. It captures the convergence and diversity of a solution set even when the optimal Pareto front (PF) is unknown. However, the high computational complexity (exponential) of the HV metric makes it impractical to be used when the number objectives are high [10-13]. Also, Pareto-surface metrics such as generational distance (GD), inverted generational distance (IGD), and Spread are simple and provide knowledge about the overall performance of the solution set [14-16]. However, these measures do not provide any insight into the distribution or spread of solution sets with respect to each objective. For example, a decisionmaker might mainly be interested in knowing real-estate properties in the range of $500,000 to $800,000. As an optimization tool provider, we need to provide the decision-maker with well distributed solutions in this price range and be able to quantitatively measure the distribution of solutions along this objective regardless of other objectives. Here, we are not implying the distribution of solutions over the Pareto-optimal surface is not important but we are suggesting the distribution of solution along each objective is equally important. Furthermore, studies have shown that these performance metrics contradict one another in the presence of extreme cases [17]. Hence, there is room for introducing more performance metrics (in conjunction with widely used performance metrics) suited for measuring the distribution of solutions along each objective. The objective of this paper is to introduce a powerful visualization method, called 3-dimensional radial coordinate with antenna (3D-RadVis Antenna) that allows MaOO researchers, decision makers and any interested party to better understand the optimization process along with intermediate and final results of an algorithm. 3D-RadVis Antenna is an extension of 3D-RadVis [6] that permits a decision maker to visually explore many-objective solution sets and identify one or more preferred optimal solutions (not solely based on the convergence and distribution of solution on the Pareto-optimal front but also distribution of solutions along each objective). As 3D-RadVis Antenna maps M-dimensional objective to a 3-dimensional space, decision makers can benefit from immersive virtual reality (VR) technologies, such as the CAVE [18] to visualize high-dimensional decision and solution space and select preferred solutions with ease. VR tools have been widely used in several disciplines where past visualization technologies are limited when analyzing and interacting with data [19-22]. In the optimization field, decision makers can use VR tools to visualize and interactively select the ideal solution according to their specific situation (set of requirements, budget etc.). In the same fashion, researchers can utilize 3D-RadVis Antenna in conjunction with VR tools to investigate aspects of many-objective optimization algorithm s (MaOOA) behavior such as, performance comparison, parameter specifications, and maybe even develop efficient algorithms to tackle MaOO problems. Also, a proper visualization tool can potentially lead to the development of effective interactive optimization methods. In conjunction with the proposed visualization approach, two performance measures, called objective-wise inverse generational distance (ObjIGD) and line distribution ( ), are proposed to quantitatively assess the convergence, spread, and diversity of solution sets obtained by many-objective population-based algorithms. The ObjIGD measure measures the convergence and distribution of a solution set for a given objective, whereas the measure measures the diversity and spread of approximate solutions without the Pareto-optimal set. Experimental results show that good distribution on the Pareto-surface does not necessarily imply a good distribution along each objective. The rest of the paper is organized as follows. Section II provides a survey of visualization methods and performance measures used in MaOO. Section III provides the technical description of the proposed visualization scheme, 3D-RadVis Antenna, along with two performance measures to measure objective-wise convergence, spread, and diversity of approximate solutions. Section IV presents experimental investigation of 3D-RadVis Antenna and the proposed measures on well-known many-objective benchmark problems and algorithms. Concluding remarks are provided in Section V. 2 II. SURVEY OF VISUALIZATION METHODS AND PERFORMANCE MEASURES USED IN MANY-OBJECTIVE OPTIMIZATION As the number of objectives increases, visualization of the approximation set becomes progressively challenging. Moreover, the applicability of quantitative metrics capable of measuring the convergence and diversity of solution sets are problematic due to inconsistencies among them [17]. In Section A we describe the classical and recent advances in visualization techniques, and in Section B we discuss performance metrics used in MaOO.

4 A. Visualization Methods 1) Classical Visualization Methods In MaOO, when the number of objectives are two or three, many effective visualization tools are available. The issue arises when the number of objectives are four or more leading to very challenging visualization of approximation sets. For instance, parallel coordinates [2] Heatmap plots [23], self-organizing maps [3] and radial coordinate visualization [5] are classical visualization tools that can be used to visualize the 3 (a) Parallel coordinates plot (b) Heatmap (c) Scatter plot matrix (d) Bubble chart (e) Self-organizing maps (f) RadVis Fig. 1. Visualization schemes used in MaOO problems showing well didtributed four-dimesional concave Pareto front generated using the equation distribution, range, and trade-off among solutions of many-objective solution sets. Nonetheless, these tools usually fail to preserve the shape or dominance relationship and are not capable of showing the convergence trend of the solution set. Parallel coordinates plot is a popular way to visualize the distribution, range, and trade-off among solutions of multi-objectives [24, 25]. Here, an objective is represented by a polyline with vertices on parallel axes placed along the x-axis. The parallel axes are equidistant vertical bars along the x-axis for each of the objectives. The y-axis corresponds to the range of possible values for each of the objectives. Despite the inability of parallel coordinate plots of showing the shape of the Pareto front, they are simple to construct, scale well to larger numbers of objectives, and are a great visualization tool to illustrate dependencies among objectives without the loss of data in the representation [26]. Fig. 1(a) depicts a parallel coordinates plot of four-dimensional concave data points generated using the equation. Similar to the parallel coordinates plot, Heatmap [23] plots represent objective values using colors as opposed to polylines with vertices. These plots are very easy to construct and can scale well to visualize higher dimensional objectives. Additionally, heat maps can show dependencies among objectives without the loss of data in the representation. However, these plots do not scale well when the number of solutions are large because the number of colors used to represent each solution is also large. Furthermore, they cannot show the shape of the PF. Fig. 1(b) depicts a Heatmap plot of four-dimensional concave data points. A scatter plot matrix is a simple visualization method capable of showing the pair-wise relationship of objectives while also preserving some information on the shape of the PF. Given an M-objective data set, a scatter plot matrix plots all objective pairs [27, 28]. However, as the number of objectives increases, the scatter plot matrix doesn't scale well because it requires a large space to show the relationship among pairs of objectives. Fig. 1(c) depicts a scatter plot matrix of four-dimensional concave data points. Bubble chart is another classical visualization method where the first and second objective values are represented using bubbles along the - and -axis and the third objective is represented by varying the size of the bubbles. There exists also a

5 variation of bubble chart that utilizes the -axis and colored bubbles to represent the 4 th and 5 th dimension [29, 30]. Fig. 1(d) depicts bubble chart representing four-dimensional concave data points. Self-organizing maps (SOM) [3] are one type of artificial neural networks (ANN) trained using unsupervised learning in order to provide a mapping from M-dimensional objective to a lower dimensional space (typically 2-D) [31]. These maps consist of nodes (neurons) associated with a weight vector of the same dimension as the input data vectors or neurons. These nodes are arranged in a 2D space using a hexagonal or rectangular grid. Typically, SOMs use the unified distance matrix (U-Matrix) [32] to store each node s average distance to its closest neighbors (different colors are used to represent each node s distance to adjacent nodes). Clusters of similar neurons are represented with light areas while dark areas indicate cluster boundaries. Fig. 1(e) depicts self-organizing map plot of four-dimensional concave data points. Radial coordinate visualization (RadVis) [5] is an alternative high-dimensional visualization method mainly used to visualize hierarchical density clusters by mapping M-dimensional data set to a 2-dimensional space using a nonlinear mapping. To better understand RadVis, consider a point in 2-dimensional space connected to M equally spaced points on a circle with springs, where each dimension value is equal to the spring constant for the corresponding spring. Now, imagine that the 2-dimensional point is allowed to move and reach equilibrium, the location of this point will be the mapping of M-dimensional data points onto a 2- dimensional space. Fig. 1(f) depicts RadVis plot of well distributed four-dimensional concave data points. Despite RadVis plots incapability of showing the shape and convergence of the PF, RadVis plots are simple to construct, scale well to large numbers of objectives, and are a great visualization tool to show the distribution of solutions. 2) Recently Proposed Visualization Methods Recently a number of visualization methods have been proposed to deal with higher dimensional data sets. The following paragraphs will introduce the main ones. Tusar and Filipic [8] proposed a visualization method that uses projection of a chosen subsection of the solution set to visualize 4-D approximation. This method allows researchers and decision makers to view the shape, range and distribution of large approximation sets. In some cases it preserves the Pareto dominance relation and the convergence of the Pareto optimal front. The drawback of this method is that it cannot scale for higher dimensions (greater than four objective). He and Yen [7] proposed another method for visualizing high dimensional objectives by mapping them onto a two-dimensional polar coordinate. Their visualization method takes each individual high-dimensional Cartesian point and assigns a radial and angular coordinate value. The radial coordinate value represents the convergence and shape of the Pareto front and the angular coordinate represents the distribution among the individuals. However, their method fails to show the relative location of a solution with respect to each objective. Walker et al. [4] proposed visualization for mutually non-dominating solution sets by using the rank solutions on each objective. As a result they enhanced the Heatmap plot by spectral seriation of both the objectives and the solutions in order to place similar objectives and similar solutions together. However, even though their scheme is able to enhance the Heatmap plot, the visualization of dominance relationships between solutions was not geometrically apparent. Recently, Ibrahim et al. [6] proposed 3D-RadVis visualization method to visualize many-objective solution sets. This method is capable of mapping M-dimensional objective space to a 3-dimensional radial coordinate plot while seeking to preserve the relative location of solutions, shape of the Pareto front, distribution of solutions, and convergence trend of an optimization process. They have used the radial coordinate system to preserve the distribution and relative location of solutions and the orthogonal distance of each solution to a reference hyper-plane to preserve the shape and convergence of a solution set. B. Performance Metrics Unlike single-objective optimization, where the assessment of the performance of a metaheuristic requires observing the best value given by an algorithm, this is not applicable in multi-objective optimization. An approximation set to the optimal PF of the problem is computed. Here, the required properties are convergence (i.e., how close the solution is to the true PF) and uniform diversity (i.e., solutions that exhibit uniform distribution). Ideally, we are interested in quality indicators that do not require the true Pareto optimal front, but are however capable of measuring the convergence and diversity of a known solution. Several indicators for measuring the aforementioned properties have been proposed in the literature. These include: capacity metrics such as the overall non-dominated vector generation (ONVG) [33] and error ratio (ER) [34], convergence metrics such as metric GD [14] and -indicator [9], diversity metrics such as the overall Pareto spread (OS) [35] and spread/diversity ( ) [16] and finally convergence-diversity metrics such as IGD [14, 15] and HV. Table I lists some MaOO performance indicators and their intended measure. In this paper we focus on three widely used performance metrics (IGD, HV, and spread) to evaluate the consistencies and contradictions among these metrics and proposed metrics. 1) Spread The spread indicator [16] measures the distribution of solutions using the extreme points of the optimal Pareto-front ( ); and is defined as follows: (1) 4

6 Where and are the Euclidean distances to the extreme solutions of the optimal Pareto front ( in the objective space; is an approximate solution set, is the Euclidean distance between consecutive solutions [17] and is the mean of these distances. A Spread value of zero indicates an ideal distribution (i.e., uniformly distributed solutions in the Pareto front). Nonetheless, the Spread indicator is based on calculating the distance between two consecutive solutions which only works for two-objective problems. This indicator can further be extended by calculating the distance from a given point to its nearest neighbours; this is based on the metric proposed in [36]: (2) where ( ) are extreme solutions in, is the number of objectives,, is an approximate solution set, is the set of Pareto optimal solutions and 5 ( ) (3) is the mean of these. 2) Inverse Generational Distance (IGD) The IGD metric [14, 15] measures the convergence and the diversity of the obtained Pareto-optimal front by measuring the distances between each solution composing and the closest solution in the obtained front. The IGD metric is defined as follows: (4) Where is the and is the obtained front. The smallest IGD value indicates a superiority of an algorithm. 3) Hypervolume (HV) The HV indicator [9] measures the volume of the dominated portion of the objective space. The interest in this indicator stems from the fact that it contains the strict Pareto compliance which is a highly desirable feature. In other words, if strictly dominates, then the HV value of is higher than the HV value of. The HV measure is obtained by computing the volume of the non-dominated set of solutions,. For every solution, a hypercube is generated with a reference point and the solution as its diagonal corner. The reference point can be generated by building a vector of worst possible objective function values. Then, the HV is computed as a union of all hyper-cubes as follows: (5) One good feature of HV is that, it does not require the to compute the volume. However, the computational complexity of HV increases exponentially with the number of objectives. A solution set with the largest HV value indicates the superiority. TABLE I. SUMMARY OF PERFORMANCE METRICS USED IN MANY-OBJECTIE OPTIMIZATION Intended measure Optimal Quality Indicator Pareto front Convergence Diversity Required Hypervolume (HV) Yes Yes No Error Ratio (ER) Yes No Yes Spread No Yes Yes Inverted Generational Distance (IGD) Yes Yes Yes R2 No Yes Yes/No Epsilon ( ) Yes No No III. PROPOSED METHODS In this section, we describe the proposed visualization technique (3D-RadVis Antenna) and two performance measures capable of quantifying the spread and distribution of MaOO solutions along each objective. The 3D-RadVis Antenna method is an extension of recently proposed 3D-RadVis [6] method where poles are added for each objective to visualize the spread and distribution of solutions along each objective.

7 A. Proposed Visualization (3D-RadVis Antenna) The framework of the proposed 3D-RadVis Antenna visualization scheme is similar to 3D-RadVis (the bottom portion of the plot); however, 3D-RadVis Antenna incorporates poles for each objective to show objective-wise distribution of solutions (the top portion of the plot). In this section, first, we explain the mapping process utilized in 3D-RadVis along with minor modification proposed to simplify the construction of the reference hyper-plane. Second, we introduce the proposed extension to 3D-RadVis. From this point forward solution set refers to a non-dominated solution set. Consider non-dominated solutions, where is the number of solutions and is the dimension of the solution; the mapping process of 3D-RadVis involves two main steps: first, determining the distance of each solution from a reference hyperplane, and second, mapping the location of -dimensional solutions to a 2-dimensional plane. The reference hyper-plane in 3D-RadVis is constructed using M extreme/boundary solutions extracted from the solution set. However, in 3D-RadVis Antenna, the reference hyper-plane for M-dimensional problem is constructed using points containing. The choice of these points is to help us standardize the 3D-RadVis Antenna plots so the same reference hyper-plane is used to compute the orthogonal distance ( ) between a point (solution) and the reference hyper-plane. Moreover we can avoid unnecessary computation and extreme situations (e.g. the number of available solutions are less than the number of objectives) when finding M boundary solutions. The orthogonal distance between points to the reference hyper-plane helps us preserving the shape and convergence of the solution set. This procedure is presented in Algorithm 1, lines 1 to 9. Next, 3D-RadVis utilizes the RadVis [5] scheme to map an -dimensional normalized solution set onto a 2-dimensional plane (. Given normalized non-dominated solutions (, where is the number of objectives and is the number of solutions, and solution can be mapped to a 2D radial space as follows: (6) and (7) where is the angular position on the circle corresponding to dimension. The RadVis mapping procedure is described in Algorithm 2. Algorithm 1: 3D-RadVisAntenna ( ) Procedure Input: matrix formed by non-dominated solutions, where is the number of objectives. Output: transformation matrix for 3D-RadVis visualization and transformation matrix for Antenna plot. // 3D-RadVis transformation 1: //Construct a reference hyper-plane = eye(m) % 2: //Calculate the normal vector for the reference hyperplane with boundary points, : norm 3: Calculate the hyper-plane equation constant, for this plane: 4: for to Calculate the perpendicular distance from n to solution : 5: end for 6: Normalize by each objective: normalize( ) 6

8 7 7: Map to 2D radial coordinates [ ] = RadVis ( ) 8: % the value of ranges from 0 to max( ). // Antenna transformation 9: Compute the highest value : = max( ). 10: Compute and location of the boundary points: [ ] = RadVis ( ) Compute the location of antenna directors (tick marks) along each objective: 11: for to 12: for to 13: 14: end for 15: end for The top portion of 3D-RadVis Antenna plot consists poles (antenna) to show the distribution of solutions along each objective. The first step to plot the antenna is to find the location of antenna poles (vertical lines). The location of these vertical lines are computed using the boundary points. The length of these vertical lines are kept at the maximum perpendicular distance of points from the reference hyper-plane ( ). This strategy will keep the top and the bottom portions of the plot with equal height. The next step is to compute the location of antenna directors (tick marks) along the vertical poles. The location of the antenna directors along each objective calculated by multiplying by and shifting these points by to place them on the top portion of the plot. The Antenna plot procedure is presented in Algorithm 1, lines 10 to 16. Fig. 2 illustrates the 3D-RadVis Antenna transformation process. Figs. 4 to 8 show 3D-RadVis Antenna plots for two- to five-objective concave, convex, mixed, disconnected, and biased Pareto surfaces. B. Proposed Performance Measures Algorithm 2: RadVis ( ) Procedure So far the focus of almost all multi- or many-objective Input: normalized non-dominated solutions, performance measures are on the convergence and distribution of where is the number of objectives and is the solutions in the objective space (Pareto-surface). Pareto-surface number of solutions. metrics such as IGD, HV, and Spread are simple and provide Output: non-linear radial coordinates mapping of knowledge about the overall performance of the solution set. However, these measures do not provide any insight into the 1: for to distribution or spread of solution set with respect to each 2: Calculate 2D radial location/mapping of objective. Further, studies have shown that these performance normalized objective: Eqs.(6) and (7) metrics contradict one another in the presence of extreme cases. 3: end for To the best of our knowledge there is no performance measure 4: capable of evaluating solution superiority based on the accuracy and distribution of solution along each objective. Therefore, in conjunction with the 3D-RadVis Antenna we propose two new performance measures called objective-wise inverse generational distance (ObjIGD) and line distribution ( ) to specifically assess the convergence, spread and diversity of MaOEAs along each objective. For example, consider a convex Pareto-surface containing 91 solutions as depicted in Fig. 3. From the scatter plot Fig. 3 (a), we see that the solution set is well-distributed on the optimal PF surface. However, when we look at the 3D-RadVis Antenna plot in Fig.3 (b) we see that the distribution of solutions along each objective is very poor. In fact, there is no solution containing,which means more than 40% of possible solutions are not accounted for by each objective. Therefore, we need to shift our research focus to find algorithms that not only provide accurate and well distributed solutions on the Pareto-front surface but also well-distributed solution along each objective and are able to measure the performance of these algorithms quantitatively. Now, let s define two such measures capable of measuring the convergence and distribution of solutions along the objective.

9 8 Fig. 2. Illustration of 3D-RadVis Antenna transformation (a) (b) Fig. 3. Scatter and 3D-RadVis Antenna plots of three-objective convex PF containing 91 solutions. (a) Scatter plot showing more than 40% of associated values. (b) 3D-RadVis Antenna showing poor objective-wise distribution. with no 1) Objective-wise Inverse Generational Distance (ObjIGD) The ObjIGD measure evaluates the convergence and distribution performance of MaOOA specifically along each objective. The main idea of ObjIGD is similar to the IGD metric, however ObjIGD measures the distance between the and the closest solution based on individual objectives. The ObjIGD for the objective is defined as follows:

10 9 ( ) (8) Where is the reference ( ), S is the approximate PF, ( ) is the solution of objective and is an approximate solution of objective. The overall ObjIGD measure is defined as: (9) where is the objective ObjIGD value and is the number of objectives. Lower value of measure implies better convergence and distribution along the objective. 2) Line Distribution ( ) The measure measures the diversity and spread of approximate solutions without the need for the. Let be the mid-points of equally divided intervals in the range of [0, 1] ([ ] [ ] [ ]), where is the number of solutions in approximate the PF, then the objective line distribution ( ) is defined as: where is a normalized approximate solution of objective. A zero value of the objective line distribution signifies uniform distribution of the approximate PF along the objective. The overall line distribution measure is defined as: (10) TABLE II. BENCHMARK TEST PROBLEMS Problem Characteristics DTLZ1 Linear, multimodal DTLZ2 Concave Convex DTLZ2 Convex DTLZ3 Concave DTLZ4 Concave, degenerate when ( ) DTLZ7 Disconnected, convex and mixed convexity WFG1 Convex, mixed, biased TABLE III. TEST NUMBER OF SOLUTIONS IN THE REFERENCE PFS AND OPTIMAL PFS. IS THE NUMBER OF OBJECTIVES AND IS NUMBER OF DIVISIONS ALONG EACH OBJECTIVE. M Reference PFs Optimal PFs TABLE IV. PARAMETER SETTINGS FOR GDE3, NSGA-II, AND NSGA-III. IS THE NUMBER OF VARIABLES Parameters GDE3 NSGA-II NSGA-III Elite- NSGA-III SBX probability ( Polynomial mutation ( - Crossover Distribution Index Mutation Distribution Index

11 10 Parameters GDE3 NSGA-II NSGA-III Elite- NSGA-III Mutation probability Crossover probability TABLE V. NUMBER OF REFERENCE POINTS AND POPULATION SIZES USED IN NSGA-III. Number of Divisions Reference Population Objectives (M) Outer Inner Points(H) Size (N) (11) where is the objective ObjIGD value and is the number of objectives. IV. EXPERIMENTAL INVESTIGATION In this section, we describe selected test benchmark problems, algorithms and their parameter settings used in this study. A. Test Problems In order to investigate the visualization capability of 3D-RadVis Antenna and the proposed performance measures, we have used five scalable MaOO benchmark problems with linear, concave, and convex shapes and two benchmark problems with complicated Pareto fronts. The benchmark problems are: DTLZ1, DTLZ2, convex DTLZ2, DTLZ3 and DTLZ4, DTLZ7 and WFG1 [37, 38]. The number of variables are ( ), where is the number of objectives and for DTLZ1, for DTLZ7, and for the remaining DTLZ test problems. The corresponding Pareto-optimal fronts lie in for the DTLZ1 problem and in for DTLZ2 DTLZ4 problems. The DTLZ7 test problem has disconnected Pareto optimal fronts, where the Pareto optimal front consists of convex and some mixed concavity. The WFG1 has a mixed Pareto-optimal front and the Pareto-optimal fronts lie in. Table II presents detailed characteristics of the test problems utilized in this study. Also, to investigate the consistencies and contradiction of the proposed performance measures and other well-known performance metrics (spread, IGD, and HV), we have constructed symmetric and continuous PFs as described in [17]. The true PFs for these experiments are constructed using: (12) Where the objectives are normalized in the range [0, 1], and is the parameter to control the geometrical shapes of PFs. To obtain the reference PFs ( ) required by the IGD and spread metrics, first we systematically generate weight vectors using the λ method or also known as the simplex lattice design [39, 40], where, are weight vectors and. The weight vectors are taken values from { } where is the number of divisions along each objectives. The number of weight vectors for objectives is then given by. Finally, the intersection point between line and (12) denotes a reference solution. We also constructed two optimal solution sets ( and ) with different diversities, where is generated based on the simplex lattice design and is generated using the Pareto-adaptive weight vectors ( method) [41] to maximize the HV value of. The reference set for HV metric is generated as = {(1, 1)}, {(1, 1, 1)}, {(1, 1, 1, 1), (1, 1, 1, 1, 1)} on 2-, 3-, 4-, and 5-D PFs, respectively. Table III presents the number of solutions in the reference PFs ( ) and optimal PFs ( and ) used in this study. (a) (b) (c) (d)

12 11 (e) (f) (g) (h) Fig. 4. Scatter and 3D-RadVis Antenna plots for two-objective PFs. and points are generated using the simplex lattice design and methods respectively. (a) Scatter plot of points for concave PF. (b) Scatter plot of points for concave PF. (c) Scatter plot of points for convex PF. (d) Scatter plot of points for convex PF. (e) 3D-RadVis Antenna plot of points for concave PF. (f) 3D-RadVis Antenna plot of points for concave PF. (g) 3D-RadVis Antenna plot of points for convex PF. (h) 3D-RadVis Antenna plot of points for convex PF. B. Algorithms and Parameter Setting In order to investigate how the newly proposed performance measures are comparable with widely used MaOO performance metrics, we have chosen a recently proposed reference-point-based algorithm and two of the earliest multi-objective evolutionary algorithms. These algorithms are: reference-point-based non-dominated sorting genetic algorithm (NSGA-III) [31], nondominated sorting genetic algorithm II (NSGA-II) [16], and generalized differential evolution generation 3 (GDE3) [42]. For detailed information on these algorithms, readers are advised to refer to the original manuscripts. Table IV presents parameter settings used in these algorithms and Table V presents the number of reference points ( ), the population size ( ), and the number of inner and outer divisions used in NSGA-III for different dimensions of test problems. Population sizes for GDE3, NSGA-II, and NSGA-III are kept the same for all test problems. C. Proposed Visualization (3D-RadVis Antenna) Here, we investigate how well 3D-RadVis Antenna maps to 2-, 3-, 4-, and 5-objective PFs to a 3-dimensioanl space. Note that since 2-objective data points are mapped to only the x-axis, the value is set to zero. 1) Visualization of Concave and Convex PFs Experimental series 1 these experiments investigate how well 3D-RadVis Antenna maps optimal solution sets ( and ) having different diversities to a 3D space. The number of solutions in and are set to 100, 153, 220, 330 for 2-, 3-, 4- and 5- objective respectively. Fig. 4 (a) to (d) show 2D scatter plots of and points for 2D concave ( ) and convex ( ) PFs. From these figures we see that has better spread on concave PF than and worse spread on convex PF. The 3D-RadVis Antenna in Fig. 4 (e) to (h) also show similar distribution of and solutions on concave and convex PF surfaces. Moreover the 3D-RadVis Antenna shows the distribution of and solutions along each individual objectives. From Fig. 4 (a) and (b) we see that the shape of the PF is similar to a quarter circle with radius = 1, centered at (0, 0). The largest distant point from the reference hyper-plane is located at the center of the arc. From the 3D-RadVis Antenna plots in Fig. 4 (g) and (f) we also see that the shape and the relative location of solutions are well-preserved. Similarly, from Fig. 4 (c) and (d) we see that the lowest distant point from the reference hyper-plane is located at the center of the arc and from the 3D-RadVis Antenna plots (Fig. 4 (g) and (h)) we see that the shape and distribution of solutions are well preserved.

13 12 (a) (b) (c) (d) (e) (f) (g) Fig. 5. Scatter and 3D-RadVis Antenna plots for three-objective PFs. and points are generated using the simplex lattice design and methods respectively. (a) Scatter plot of points for concave PF. (b) Scatter plot of points for concave PF. (c) Scatter plot of points for convex PF. (d) Scatter plot of points for convex PF. (e) 3D-RadVis Antenna plot of points for concave PF. (f) 3D-RadVis Antenna plot of points for concave PF. (g) 3D-RadVis Antenna plot of points for convex PF. (h) 3D-RadVis Antenna plot of points for convex PF. (h)

14 13 (a) (b) (c) Fig. 6. 3D-RadVis Antenna plots for four-objective PFs. and points are generated using the simplex lattice design and methods respectively. (a) points for concave PF. (b) points for concave PF. (c) points for convex PF. (d) points for convex PF. (d) Experimental series 2 these experiments investigate how well 3D-RadVis Antenna maps 3-objective optimal solution sets ( and ) with different diversities onto 3D space. Fig. 5 (a) to (d) show 3D scatter plots of and points for 3D concave ( ) and convex ( ) PFs. Similar to the previous experiment 3D-RadVis Antenna is able to capture the shape and distribution of 3D PFs. Moreover, 3D-RadVis Antenna is clearly able to show the distribution of solutions along each objective which would not be possible using other types of visualization tools. For example, in Fig. 5 (g), for the solution set, we see sparse region at the top portion of the antenna and high dense values at the bottom of the antenna. In other words, many of the solutions are concentrated on the interval of [0, 0.5] along each objective and leaving much of the interval [0.5, 1] underrepresented. However in Fig. 5 (h), for solution set, we see that the distribution of solutions along each objective is much better than that of. This is because the solution set is generated using the Pareto-adaptive weight vectors ( method) with highest possible HV value. Experimental series 3 these experiments investigate how well 3D-RadVis Antenna maps 4-objective optimal solution sets ( and ) with different diversities onto 3D space. Fig. 6 (a) to (d) show 3D-RadVis Antenna plots of and points for 4D concave ( ) and convex ( ) PFs. One interesting observation from these plots is that the distribution of the solution set along each individual objective (Fig. 6 (a)) is better than the solution set (Fig. 6 (b)) even though the solution set has higher HV value. Thus, we can conclude that a solution set with higher HV value does not necessarily exhibit better distribution of solutions along each objective. More discussion on this point is provided in subsection D.

15 Experimental series 4 these experiments investigate how well 3D-RadVis Antenna maps 5-objective optimal solution sets ( and ) with different diversities onto 3D space. Fig. 7 (a) to (d) show 3D-RadVis Antenna plots of and points for 5D concave ( ) and convex ( ) PFs. Similar to the previous three experiments we can see that 3D-RadVis Antenna not only is able to capture the shape and distribution of solutions in higher dimension but also provide us valuable information regarding the quality of approximate solutions. Experimental series 5 these experiments investigate how well 3D-RadVis Antenna maps 3- to 5-objective solutions with complicated PFs, namely DTLZ7 and WFG1. The DTLZ7 test problem has disconnected convex and some mixed convexity Pareto optimal fronts. On the other hand, the WFG1 has a mixed and biased Pareto-optimal front. Fig. 8(a), (c) and (e) depicts 3D-RadVis Antenna transformation of the DTLZ7 problem. As the number of dimension increases, the bottom portion of the 3D- RadVis Antenna plot is able to capture shape and distribution of solutions on the Pareto optimal front. Similarly, the top portion of the plot (Antenna) is also able to capture the distribution of solution along each objective. From this plot we see that the DTLZ7 test problem has good distribution of solution along the (the last) objective and very sparse distribution along the remaining ( ) objectives. Furthermore, as the number of objectives increase the sparseness of solutions along these ( ) objectives is also increases. Similarly Fig. 8(b), (d) and (e) depicts 3D-RadVis Antenna transformation of the WFG1 test problem. From these figures we see that 3-RadVis Antenna is able to capture the shape (mixed type) and the distribution of solutions along each objectives and on the Pareto optimal surface. 14 (a) (b) (c) Fig. 7. 3D-RadVis Antenna plots for four-objective PFs. and points are generated using the simplex lattice design and methods respectively. (a) points for concave PF. (b) points for concave PF. (c) points for convex PF. (d) points for convex PF. (d)

16 15 (a) Three-objective DTLZ7 (b) Three-objective WFG1 (c) Four-objective DTLZ7 (d) Four-objective WFG1 (e) Five-objective DTLZ7 (f) Five-objective WFG1 Fig. 8. 3D-RadVis Antenna plots of three- to five-objective DTLZ7 and WFG1 test problems. The DTLZ7 test problem has disconnected, convex and mixed convexity PF and the WFG1 test problem has convex, mixed and biased PF. 2) Tracking the Progress of an Optimizer Using 3D-RadVis Antenna The previous section has shown the effectiveness of 3D-RadVis Antenna to visualize and assess the quality of different solution sets when the number of objectives are two or more. In this section we investigate how 3D-RadVis Antenna can be used to plot approximate PFs or monitor the progress of an algorithm when the is known. The 3D-RadVis Antenna plots in Fig. 9 show the performance of NSGA-III algorithm for 5-objective DTLZ2 test problem after 25, 50, 200 and 400 generations. From these plots we can see that NSGA-III is able to converge to the while maintaining well distributed solutions. However, as

17 the generation progressed, the distribution of solutions along each objective deteriorated due to reference points utilized in NSGA-III algorithm. This phenomenon was expected as NSGA-III tries to guide solutions towards well-spread reference points on the Pareto-surface but ignoring the distribution of solutions along each objective. From this experiment, we can see that 3D- RadVis Antenna can effectively be used by researchers and decision makers to explore, understand and ultimately improve the search behavior of an algorithm. Furthermore, in an interactive environment it is possible to rotate and visualize solutions from different viewpoints to better understand the relationships among solutions and selectively guide solutions towards the optimal PF. Effectiveness of Proposed Performance Measures In this section we investigate the effectiveness of proposed performance measures, ObjIGD and when assessing the performance of MaOOAs. The first set of experiments investigate the similarities and differences among the proposed metrics and widely used MaOO performance metrics on PFs containing various geometrical shapes. The second set of experiments examines how the proposed visualization scheme and performance measures can be combined to provide useful information regarding the performance of MaOOA. The last set of experiments investigates the effectiveness of ObjIGD and measures when evaluating four MaOOAs on five widely used benchmark problems. 16 (a) 25 generations (b) 50 generations Fig. 9. (c) 200 generations (d) 400 generations 3D-RadVis Antenna plots showing the progress of obatained solutions by NSGA-III for 5-objective DTLZ2 test problem after 25, 50, 200, and 400 generations. 3) Comparison of Proposed Performance Measures with Spread, IGD, and HV on 2-, 3-, 4-, and 5-D PFs In this section, we investigate the similarities and differences of the proposed measures and three widely used MaOO performance metrics on various PF shapes and dimensions. Table III shows the number of solutions in the reference PFs and optimal PFs used in this experiment. The optimal PF, is generated using the simplex lattice design and is generated using method by restructuring points so that they attain the maximum HV value. Figures 10 to 13 show differences in metrics (diversity, objective-wise diversity, convergence and diversity, convergence and diversity, and objective-wise convergence and diversity

18 ) on 2-, 3-, and 4-objective PFs. A negative difference value indicates the superiority over while a positive difference value indicates the superiority over according to the assessed metric. A difference value of zero indicates no superiority between and. Experimental series 1 examines the relationship of five performance metrics on 2D PFs. Fig. 10 shows the difference values of each metric on PFs, where. When comparing the diversity metrics ( and ), in Fig. 10 (a) and (b), both metrics exhibit similar results when. However when is not able to distinguish the difference between the two optimal sets. Fig. 10 (c) to (e) show the difference values for convergence-diversity measures (HV, IGD and ObjIGD). The difference values for IGD are similar to ObjIGD, however both measures contradicting difference values with the HV metrics on concave PFs,. Experimental series 2 examines the relationship of performance measures on 3D PFs. Fig. 11 shows the difference values of each measure on PFs, where. When comparing the diversity measures ( and ), in Fig. 11 (a) and (b), the metric was unable to decisively distinguish the difference between the two optimal PFs for. On the other hand, the measure was able to distinguish s superiority on and inferiority on. Fig. 11 (c) to (e) show the difference values for convergence-diversity measures (HV, IGD and ObjIGD). Experimental series 3 examines the relationship of performance metrics on 4D PFs. Fig. 12 shows the difference values of each metric on PFs, where. As it can be seen in Fig. 12 (a) and (b), the diversity measures ( and ) exhibit similar results for the convex portion of the PF and contradictory results for the concave portion of the PF. When comparing the convergence-diversity measures (HV, IGD, and ObjIGD), in Fig. 12 (c) to (e), the IGD and ObjIGD measures show identical trend on, however the HV metric shows contradictory results for all shapes. Experimental series 4 examines the relationship of performance metrics on 3D PFs. Fig. 13 shows the difference values of each metric on PFs, where. When comparing the diversity measures ( and ), in Fig. 13 (a) and (b), the and measures show consistent trend as the previous experiment (4D PFs). Similarly, the convergence-diversity metrics (see. Fig. 13 (c) to (e)) also show the same trend as the previous experiment. However, the HV metric difference values for is close to zero and thus unable to clearly distinguish the superiority of a solution set for convex PFs. From our experiments on 2- to 5-objective PFs for the two optimal solution sets on different shapes of PFs, we summarize our observations as follows: The IGD and ObjIGD performance measures showed similar trend for all test cases, however the ObjIGD results curves were smoother than the IGD metric. Therefore, the ObjIGD measure can be used as complementary measure to IGD as a tie breaker. As the number of objectives increases, the difference HV values on convex PF, approaches zero. This means, in the presence of extreme cases, the HV metrics cannot distinguish the superiority of a solution set for high dimension on convex PFs. Therefore, in such cases the ObjIGD measure can be used to determine the superior solution set. In almost all experiments the difference values on concave PFs were unstable (the difference values fluctuated throughout this interval). However the number of dimension grow the measure was stable for all PFs and as a result this characteristics make it desirable when we need to determine a solution set with greater diversity. 4) Using the Proposed Visualization and Performance Measures to Assess MaOOAs In the previous sections, we have shown how 3D-RadVis Antenna and the proposed performance measures can be used separately. In this section, we will show how these two ideas can be combined to assess and investigate the performance of a MaOOA visually and quantitatively. Fig.s 14 to 17 show 3D-RadVis Antenna plots with and ObjIGD values along each objective for GDE3, NSGA-II and NSGA-III algorithms on 3- and 5-objective DTLZ2 and convex DTLZ2 test problems. In 3- dimesional DTLZ2 (Fig. 14), we see that GDE3 has slightly better distribution of solutions along each objective than NSGA-II. However, both NSGA-II and GDE3 have significantly better distribution in all objectives than NSGA-III. The poor performance of NSGA-III along each objective is attributed to the fact that NSGA-III tries to guide solutions towards well-spread reference points onto the optimal Pareto-surface while disregarding the distribution of solutions along each objective. When comparing the IGD values for these algorithms ( for GDE3, NSGA-II and NSGA-III respectively), NSGA-III exhibited superior performance than the NSGA-II and GDE3 as the IGD metric measures the convergence and distribution of solutions on the optimal PF surface. In 3-objective convex DTLZ2, from Fig. 15 (a) and (b), GDE3 and NSGA-II show very poor distribution on the optimal Pareto-surface while attaining far better distribution of solutions along each objective. On the other hand, NSGA-III (see Fig. 15 (c)) is able to distribute 92 solutions onto 91 uniformly placed reference points on a normalized hyper-plane. However, NSGA-III 17

19 still fails to attain good distribution of solutions along each objective. The IGD values for GDE3, NSGA-II and NSGA-III are,, and respectively. 18 (a) (b) (c) (d) (e) Fig. 10. Differences in metrics of two-objective and solutions containing 25, 50, and 100 points. and points are generated using the simplex lattice design and methods respectively and. A positive value indicates is superior to. (a) Diversity. (b) Objective-wise diversity. (c) Convergence and diversity. (d) Convergence and diversity. (e) Objective-wise convergence and diversity. (a) (b) (c) (d) (e)

20 Fig. 11. Differences in metrics of three-objective and solutions containing 45, 91, and 153 points. and points are generated using the simplex lattice design and methods respectively and. A positive value indicates is superior to. (a) Diversity. (b) Objective-wise diversity. (c) Convergence and diversity. (d) Convergence and diversity. (e) Objective-wise convergence and diversity. 19 (a) (b) (c) (d) (e) Fig. 12. Differences in metrics of four-objective and solutions containing 56, 120, and 220 points. and points are generated using the simplex lattice design and methods respectively and. A positive value indicates is superior to. (a) Diversity. (b) Objective-wise diversity. (c) Convergence and diversity. (d) Convergence and diversity. (e) Objective-wise convergence and diversity. (b) (c) (d) (e) (f)

21 Fig. 13. Differences in metrics of five-objective and solutions containing 70, 210, and 330 points. and points are generated using the simplex lattice design and methods respectively and. A positive value indicates is superior to. (a) Diversity. (b) Objective-wise diversity. (c) Convergence and diversity. (d) Convergence and diversity. (e) Objective-wise convergence and diversity. In 5-objective DTLZ2 test problem, from Fig. 16 (a), GDE3 show fair distribution on the optimal Pareto-surface while attaining an excellent distribution along each objective. On the other hand, from Fig. 16 (b), we see that NSGA-II is not able to maintain well-converged or good distribution of solutions on the optimal Pareto surface or along each objective. This shows that NSGA-II performs better on bi- and tri-objective problems and loses its power when the number of objectives is high. Fig. 16 (c) show that the NSGA-III algorithm is able to maintain well-converged and uniformly distributed solutions on the optimal PF surface and as a result it is able to achieve superior IGD score. The IGD scores for GDE3, NSGA-II and NSGA-III are,, and respectively. Similar to the previous results NSGA-III still failed to show good distribution along each objective. In the 5-objective convex DTLZ2 test problem, from Fig. 17 (b) we see that the performance of NSGA-II algorithm continues to deteriorate and as a result it showed very poor IGD, ObjIGD and scores. However, from Fig. 17 (a) and (c) we see that GDE3 and NSGA-III are able to maintain their strength (i.e. good convergence and distribution of solutions on the optimal PF surface for NSGA-III and good convergence and well-distributed solution along each objective for GDE3) for 5- objective convex DTLZ2 test problem. The IGD scores for GDE3, NSGA-II and NSGA-III are,, and respectively. An interesting observation from Figs 15 (c) and 17 (c) is that even though the supplied reference points for NSGA-III are uniformly distributed on normalized hyper-plane, the distribution of solution obtained by the NSGA-III algorithm are poor outside the intermediate region of the surface. This poor distribution of solutions also surfaced in the 3D- RadVis Antenna plots. D. Comparison of Proposed Performance Measures with Spread, IGD, and HV on 2-, 5-, and 8-D on Benchmark Test Problems In this experiment series, we compare the performance of GDE3, NSGA-II, and NSGA-III algorithms based on the IGD [43-45], ObjIGD, and measures on 3-, 5- and 8-objective DTLZ1 to DTLZ4, and convex DTLZ2 test problems. The reference Pareto fronts used in the IGD and ObjIGD measures are mathematically generated to evaluate the efficacy of the proposed measure. Table VI shows the IGD, ObjIGD and scores as well as the best, the worst, the median, and the average results for GDE3, NSGA-II and NSGA-III. The first best performing algorithm for each measure is emphasized in grey shade and the second best is emphasised in boldface. From Table VI, as expected, we see that NSGA-III has the worst score when measuring the overall distribution of solutions along each objective. This is because NSGA-III tries to guide solutions towards well-spread reference points on the Paretosurface while discounting the distribution of solutions along each objective. When using the measure, both NSGA-II and GDE3 shown comparable performance as the measure only measures the distribution of solutions along each objective while disregarding the convergence of solutions. On the other hand, when using the ObjIGD measure, GDE3 has shown superior performance as the ObjIGD measure measures the convergence and the distribution of solutions along each objective. Overall, GDE3 was the dominant algorithm when comparing the performance of algorithms in all measures utilized in this study. To improve the distribution of solutions along each objective while preserving the convergence and distribution of solutions on the optimal PF surface, we suggest using an alternate systematic way of generating reference points used in NSGA-III. For example, NSGA-III uses Das and Dennis s [46] systematic approach to generate uniformly distributed reference points on a normalized hyper-plane. Fig 18(a) shows 3D-RadVis Antenna plot of reference points generated using the Das and Dennis s approach for 5-objective with 6 divisions. From this plot we see that the distribution of these reference points along each objective is poor. Instead we can use a method similar the approach [41] (in our case and the optimal scaling parameter is calculated based on the minimum ) to adjust the reference points with better distribution along each objective. Fig. 18 (b) shows 3D-RadVis Antenna plot of adjusted reference points using the approach From our studies on the proposed measures (ObjIGD and ), we summarize our findings as follows: 1) The ObjIGD and measures are not meant to replace popular performance metrics such as IGD and HV, but rather to complement existing performance metrics. 2) The measure should be used when solely interested in measuring the distribution of solutions along each objective. 3) The ObjIGD measure should be used when mainly interested in measuring the distribution of solutions along each objective but still require some information of convergence. 4) When measuring the overall performance of a MaOOA, the ObjIGD and measures can be used as tiebreaker. 5) When designing reference-point based algorithms, we should also incorporate the idea of distribution of points along each objective. 20

22 21 (a) GDE3 (b) NSGA-II (c) NSGA-III Fig D-RadVis Antenna plots of three-objective DTLZ2 problem showing the convergence and diversity of obtained solutions onto the Pareto optimal surface and the convergenc and diversity of obtained solutions along each objective. A small value of and indicates the superiority of the solution along the objective.

23 22 (d) GDE3 (e) NSGA-II (f) NSGA-III Fig D-RadVis Antenna plots ofthree-objective convex DTLZ2 problem showing the convergence and diversity of obtained solutions onto the Pareto optimal surface and the convergenc and diversity of obtained solutions along each objective. A small value of and indicates the superiority of the solution along the objective.

24 23 (a) GDE3 (b) NSGA-II (c) NSGA-III Fig D-RadVis Antenna plots five-objective DTLZ2 problem showing the convergence and diversity of obtained solutions onto the Pareto optimal surface and the convergenc and diversity of obtained solutions along each objective. A small value of and indicates the superiority of the solution along the objective.

25 24 (a) GDE3 (b) NSGA-II (c) NSGA-III Fig D-RadVis Antenna plots five-objective convex DTLZ2 problem showing the convergence and diversity of obtained solutions onto the Pareto optimal surface and the convergenc and diversity of obtained solutions along each objective. A small value of and indicates the superiority of the solution along the objective. TABLE VI. BEST, MEDIAN, WORST, AND AVERAGE IGD, OBJIGD AND VALUES FOR GDE3, NSGA-II, AND NSGA-III ON M-OBJECTIVE DTLZ TEST PROBLEMS. FIRST BEST PERFORMING ALGORITHM IS SHOWN IN GREY HIGHLIGHT AND SECOND BEST IS SHOWN IN BOLDFACE.