New Heuristic And Metaheuristic Approaches Applied To The Multiple-choice Multidimensional Knapsack Problem

Transcription

1 Wright State University CORE Scholar Browse all Theses and Dissertations Theses and Dissertations 2008 New Heuristic And Metaheuristic Approaches Applied To The Multiple-choice Multidimensional Knapsack Problem Chaitr Hiremath Wright State University Follow this and additional works at: Part of the Engineering Commons Repository Citation Hiremath, Chaitr, "New Heuristic And Metaheuristic Approaches Applied To The Multiple-choice Multidimensional Knapsack Problem" (2008). Browse all Theses and Dissertations This Dissertation is brought to you for free and open access by the Theses and Dissertations at CORE Scholar. It has been accepted for inclusion in Browse all Theses and Dissertations by an authorized administrator of CORE Scholar. For more information, please contact

2 New Heuristic And Metaheuristic Approaches Applied To The Multiple-choice Multidimensional Knapsack Problem A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy by Chaitr S. Hiremath M.S., Wright State University, Wright State University

3 COPYRIGHT BY Chaitr S. Hiremath 2008

4 WRIGHT STATE UNIVERSITY SCHOOL OF GRADUATE STUDIES February 23, 2008 I HEREBY RECOMMEND THAT THE DISSERTATION PREPARED UNDER MY SUPERVISION BY Chaitr S. Hiremath ENTITLED New Heuristic And Metaheuristic Approaches Applied To The Multiple-choice Multidimensional Knapsack Problem BE AC- CEPTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DE- GREE OF Doctor of Philosophy. Raymond R. Hill, Ph.D. Dissertation Director Ramana Grandhi, Ph.D. Director, Engineering Ph.D. Program Committee on Final Examination Joseph F. Thomas, Jr., Ph.D. Dean, School of Graduate Studies Raymond R. Hill, Ph.D. James T. Moore, Ph.D. Xinhui Zhang, Ph.D. Gary Kinney, Ph.D. Mateen Rizki, Ph.D.

5 ABSTRACT Hiremath, Chaitr. Ph.D., Department of Biomedical, Industrial and Human Factors Engineering, Wright State University, New Heuristic And Metaheuristic Approaches Applied To The Multiple-choice Multidimensional Knapsack Problem. The knapsack problem has been used to model various decision making processes. Industrial applications find the need for satisfying additional constraints and these necessities lead to the variants and extensions of knapsack problems which are complex to solve. Heuristic algorithms have been developed by many researchers to solve the variants of knapsack problems. Empirical analysis has been done to compare the performance of these heuristics. Little research has been done to find out why certain algorithms perform well on certain test problems while not so well on other test problems. There has been little work done to gain knowledge of the test problem characteristics and their effects on algorithm performance. The research focuses on the Multiple-choice Multidimensional Knapsack Problem (MMKP), a complex variant of the knapsack problem. The objectives of the research are fourfold. The first objective is to show how empirical science can lead to theory. The research involves the empirical analysis of current heuristics with respect to problem structure especially constraint correlation and constraint slackness settings. The second objective is to consider the performance traits of heuristic procedures and develop a more diverse set of MMKP test problems considering problem characteristics like the number of variables, number of constraints, constraint correlation, and constraint right-hand side capacities. The third objective is the development of new heuristic approaches for solving the MMKP. This involves examining the existing heuristics against our new test set and using the analysis of the results to help in the development of new heuristic approaches. The fourth objective is to develop improved metaheuristic procedures for the MMKP using the improved heuristic approaches to initialize searches or to improve local search neighborhoods. iii

6 Contents 1 Introduction Discussion of Knapsack Problems Overview of the Dissertation Research Contributions of the Dissertation Research Multi-Dimensional Knapsack Problems Introduction Branch-and-Bound Approach Dynamic Programming Greedy Heuristics Transformation Heuristics Metaheuristic Approaches Tabu Search (TS) Genetic Algorithm (GA) Simulated Annealing (SA) Extensions and Variants of the Knapsack Problem involving the notion of sets Introduction Multiple Knapsack Problems (MKP) MKP Formulation Heuristic Solution Approaches for MKP Multiple Choice Knapsack Problems (MCKP) MCKP Formulation Heuristic Solution Approaches for MCKP Multiple-choice Multi-dimensional Knapsack Problems (MMKP) MMKP Formulation Heuristic Solution Approaches for MMKP Applications and Formulations of the MMKP-type problems Legacy Heuristics and Test Problems Analysis Introduction Legacy Heuristics Legacy Heuristics for Multiple Knapsack Problems (MKP) Legacy Heuristics for Multiple Choice Knapsack Problems (MCKP) iv

7 4.2.3 Legacy Heuristics for Multiple-choice Multi-dimensional Knapsack Problems (MMKP) Test Problem Analysis Test Problem Analysis for Multiple Knapsack Problems (MKP) Test Problems for Multiple Choice Knapsack Problems (MCKP) Test Problems for Multiple-choice Multi-dimensional Knapsack Problems (MMKP) Problem Structure Analysis of Test Problems Structure of MDKP Test Problems Structure of MKP Test Problems Structure of MCKP Test Problems Structure of MMKP Test Problems Summary Empirical Analyses of Legacy MMKP Heuristics and Test Problem Generation Introduction Problem Generation and Problem Characteristics for MMKP Standard MMKP Test Problem Generation Analytical MMKP Test Problem Generation Competitive MMKP Test Problem Generation Analytical MMKP Test Sets Versus Available MMKP Test Set Empirical Analyses of MMKP Heuristics on Available Test Problems Empirical Analyses of MMKP Heuristics on New MMKP Test Problem Set Analyses based on Constraint Right-Hand Side Setting Analyses based on Correlation Structure Summary New Greedy Heuristics for the MMKP Introduction A TYPE-based Heuristic for the MMKP New Greedy Heuristic Version 1 (CH1) NG V3 Heuristic (Cho 2005) CH1 Implementation Empirical Tests for the CH1 Implementation New Greedy Heuristic Version 2 (CH2) CH2 Implementation Empirical Tests for the CH2 Implementation Summary Metaheuristic Solution Procedure for the MMKP Introduction Concept of a Search Neighborhood First-Level Tabu Search (FLTS) for the MMKP FLTS Implementation Empirical Tests for the FLTS Implementation v

8 7.3.3 Extensions of the FLTS for the MMKP Sequential Fan Candidate List (FanTabu) for the MMKP FanTabu Implementation Empirical Tests for the FanTabu Implementation CPCCP with Fan Candidate List (CCFT) for the MMKP CCFT Implementation Empirical Tests for the CCFT Candidate List Implementation Comparison of TS approaches with Reactive Local Search Approach (RLS) RLS Approach Empirical Tests Comparing TS approaches with RLS Approach Summary Summary, Contributions, and Future Avenues Summary and Contributions Legacy Heuristics and Test Problem Analysis Insights on Heuristic Performance Based on Problem Structure Empirical Science Leading to Theory New Test Set Development New Greedy Heuristics Development Metaheuristic Solution Procedure Development Future Avenues Appendices 219 A Additional Results from Empirical Tests 219 B Details on Cho Generation Approach 225 Bibliography 228 vi

9 List of Figures 2.1 Different types of crossover operators ((Zalzala and Fleming 1997) and (Renner and Ekart 2003)) Genetic Algorithm Flowchart (Renner and Ekart 2003) Range of Correlation Values Between Objective Function and Constraint Coefficients for MDKP Standard Test Problems Range of Correlation Values Between Constraint Coefficients for MDKP Standard Test Problems Range of Correlation Values Between Objective Function and Constraint Coefficients for Hung and Fisk MKP Test Problems Range of Correlation Values Between Objective Function and Constraint Coefficients for Martello and Toth MKP Test Problems Range of Correlation Values Between Objective Function and Constraint Coefficients for Sinha and Zoltners MCKP Test Problems Range of Correlation Values Between Objective Function and Constraint Coefficients for Moser s MMKP Test Problems Range of Correlation Values Between Objective Function and Constraint Coefficients for Khan s MMKP Test Problems Range of Correlation Values Between Constraint Coefficients for Khan s MMKP Test Problems Range of Correlation Values Between Objective Function and Constraint Coefficients for the correlated MMKP Test Problems generated from Khan s MMKP Test Problem Generator Range of Correlation Values Between Constraint Coefficients for the correlated MMKP Test Problems generated from Khan s MMKP Test Problem Generator Range of Correlation Values Between Objective Function and Constraint Coefficients for Available MMKP Test Problems Range of Correlation Values Between Objective Function and Constraint Coefficients for the generated MMKP Test Problem with 5 classes, 10 items, 5 knapsacks Range of Correlation Values Between Objective Function and Constraint Coefficients for New MMKP Test Sets Range of the Right-Hand Side Values of the Knapsack Constraints of Available MMKP Test Problems Range of the Right-Hand Side Values of the Knapsack Constraints of Analytical MMKP Test Sets vii

10 6.1 Flowchart for TYPE-based Heuristic Comparison of TYPE-based Heuristic according to Problem Type Flow Chart of NG V3 Heuristic (Cho 2005) Comparison of CH1 based on Number of Problems Solved to Optimal on New MMKP Test Sets Comparison of CH1 based on Number of Times Equal to Best on New MMKP Test Sets Comparison of All Heuristics based on Number of Problems Solved to Optimal on New MMKP Test Sets Comparison of All Heuristics based on Number of Times Equal to Best on New MMKP Test Sets Comparison of All Heuristics based on Percentage Relative Error on New MMKP Test Sets Comparison of FLTS based on Number of Problems Solved to Optimal on New MMKP Test Sets Comparison of FLTS based on Number of Times Equal to Best on New MMKP Test Sets Sequential Fan Candidate List (Glover and Laguna 1997) Comparison of FanTabu based on Number of Problems Solved to Optimal on New MMKP Test Sets Comparison of FanTabu based on Number of Times Equal to Best on New MMKP Test Sets Comparison of CCFT based on Number of Problems Solved to Optimal on New MMKP Test Sets Comparison of CCFT based on Number of Times Equal to Best on New MMKP Test Sets Comparison of All Heuristics based on Number of Problems Solved to Optimal on New MMKP Test Sets Comparison of All Heuristics based on Number of Times Equal to Best on New MMKP Test Sets Comparison of All Heuristics based on Percentage Relative Error on New MMKP Test Sets A.1 Range of Correlation Values Between Objective Function and Constraint Coefficients for the generated MMKP Test Problem with 10 classes, 10 items, 5 knapsacks 222 A.2 Range of Correlation Values Between Objective Function and Constraint Coefficients for the generated MMKP Test Problem with 25 classes, 10 items, 5 knapsacks 222 A.3 Range of Correlation Values Between Objective Function and Constraint Coefficients for the generated MMKP Test Problem with 5 classes, 10 items, 10 knapsacks 223 A.4 Range of Correlation Values Between Objective Function and Constraint Coefficients for the generated MMKP Test Problem with 10 classes, 10 items, 10 knapsacks223 A.5 Range of Correlation Values Between Objective Function and Constraint Coefficients for the generated MMKP Test Problem with 25 classes, 10 items, 10 knapsacks224 viii

11 List of Tables 4.1 Factors and Measures used in Empirical Analysis of the MKP Heuristics Factors and Measures used in Empirical Analysis of the MCKP Heuristics Factors and Measures used in Empirical Analysis of the MMKP Heuristics Theoretical and Practical Objective and Constraint Coefficient Correlation for Pisinger s MKP Test Problems Theoretical and Practical Objective and Constraint Coefficient Correlation for Pisinger s MCKP Test Problems Correlation Between Objective and Constraint Coefficients Analysis of Khan s MMKP Test Problems Interconstraint Correlation Coefficients Analysis of Khan s MMKP Test Problems Coded Combinations of Slackness Settings for 5 Knapsacks Coded Combinations of Slackness Settings for 10 Knapsacks Coded Combinations of Slackness Settings for 25 Knapsacks Correlation Between Objective and Constraint Coefficients Analysis of Available MMKP Test Problems Correlation Between Objective and Constraint Coefficients Analysis of New MMKP Test Sets Right-Hand Side Analysis of the Knapsack Constraints in Available MMKP Test Problems Right-Hand Side Analysis of the Knapsack Constraints of Analytical MMKP Test Sets Reported Results of the Legacy MMKP Heuristics on the 13 Available Test Problems Obtained Results of the Legacy MMKP Heuristics on the 13 Available Test Problems Performance Summary of the Legacy Heuristics on Khan s 13 Available Test Problems Percentage Relative Error of Legacy Heuristics on Khan s 13 Available Test Problems Results of the Legacy MMKP Heuristics on the Correlated Test Problems Generated using Khan s Test Problem Generator Performance Summary of the Legacy Heuristics on Extra Generated Test Problems Percentage Relative Error of Legacy Heuristics on Extra Generated Test Problems Number of times best heuristic for each of the Right-Hand Side Combinations for 5 classes, 10 items, 5 knapsacks Number of times best heuristic for each of the Right-Hand Side Combinations for 10 classes, 10 items, 5 knapsacks Number of times best heuristic for each of the Right-Hand Side Combinations for 25 classes, 10 items, 5 knapsacks ix

12 5.18 Number of times best heuristic for each of the Right-Hand Side Combinations for 5 classes, 10 items, 10 knapsacks Number of times best heuristic for each of the Right-Hand Side Combinations for 10 classes, 10 items, 10 knapsacks Number of times best heuristic for each of the Right-Hand Side Combinations for 25 classes, 10 items, 10 knapsacks Number of times best heuristic for each of the Right-Hand Side Combinations for 5 classes, 10 items, 25 knapsacks Number of times best heuristic for each of the Right-Hand Side Combinations for 10 classes, 10 items, 25 knapsacks Number of times best heuristic for each of the Right-Hand Side Combinations for 25 classes, 10 items, 25 knapsacks Hypothesis Test Results of the Influence of Correlation Structure on Heuristic Performance Hypothesis Test Results of the Influence of the Correlation Structure on Heuristic Performance based on Constraint Tightness for MMKP problems with 5 knapsacks, 10 items, and varied classes of 5, 10, and Hypothesis Test Results of the Influence of the Correlation Structure on Heuristic Performance based on Constraint Tightness for MMKP problems with 10 knapsacks, 10 items, and varied classes of 5, 10, and Hypothesis Test Results of the Influence of the Correlation Structure on Heuristic Performance based on Constraint Tightness for MMKP problems with 25 knapsacks, 10 items, and varied classes of 5, 10, and MMKP Problems with 5 Knapsacks, Different Right-Hand Side Combinations, and Best Heuristic MMKP Problems with 10 Knapsacks, Different Right-Hand Side Combinations, and Best Heuristic Number of Times Equal to Best of the Legacy Heuristic Solutions according to Problem Type Solution Quality for CH1 on Khan s 13 Available Test Problems Performance Summary of the CH1 with Legacy Heuristics on Khan s 13 Available Test Problems Percentage Relative Error by CH1 and each Legacy Heuristic for the Khan s 13 Available Test Problems Solution Quality for CH1 on Additional Test Problems generated via Khan s Approach Performance Summary of the CH1 with Legacy Heuristics on Additional Test Problems generated via Khan s Approach Performance Summary of the CH1 with Legacy Heuristics on New MMKP Test Sets Percentage Relative Error by CH1 and each Legacy Heuristic for the New MMKP Test Sets Computational Time in milliseconds by CH1 and each Legacy Heuristic for the New MMKP Test Sets Solution Quality for CH2 on Khan s 13 Available Test Problems Performance Summary of All Heuristics on Khan s 13 Available Test Problems Percentage Relative Error of All Heuristics on Khan s 13 Available Test Problems. 149 x

13 6.15 Solution Quality of All Heuristics on Additional Test Problems Performance Summary of the CH2 against Legacy Heuristics on Additional Test Problems Performance Summary of the CH2 with Legacy Heuristics on New MMKP Test Sets Percentage Relative Error of All Heuristics for the New MMKP Test Sets Computational Time in milliseconds of All Heuristics on New MMKP Test Sets Comparison of All Heuristics based on Percentage of Optimum on the Khan s MMKP Test Sets Comparison of All Heuristics based on Percentage of Optimum on the New MMKP Test Sets Solution Quality for FLTS on Khan s 13 Available Test Problems Performance Summary of the FLTS with Legacy Heuristics on Khan s 13 Available Test Problems Percentage Relative Error by FLTS and each Legacy Heuristic for the Khan s 13 Available Test Problems Solution Quality for FLTS on 24 Additional Test Problems Performance Summary of the FLTS with Legacy Heuristics on 24 Additional Test Problems Performance Summary of the FLTS with Legacy Heuristics on New MMKP Test Sets Percentage Relative Error by FLTS and each Legacy Heuristic for the New MMKP Test Sets Computational Time in milliseconds by FLTS and each Legacy Heuristic on the New MMKP Test Sets Solution Quality for FanTabu on Khan s 13 Available Test Problems Performance Summary of the FanTabu and Other Heuristics on Khan s 13 Available Test Problems Percentage Relative Error by FanTabu and Other Heuristics for the Khan s 13 Available Test Problems Solution Quality for FanTabu and Other Heuristics on 24 Additional Test Problems Performance Summary of the FanTabu with Other Heuristics on 24 Additional Test Problems Performance Summary of the FanTabu with Other Heuristics on New MMKP Test Sets Percentage Relative Error by FanTabu and Other Heuristics for the New MMKP Test Sets Computational Time in milliseconds by FanTabu and Other Heuristics on the New MMKP Test Sets Solution Quality for CCFT on Khan s 13 Available Test Problems Performance Summary of the CCFT with Other Heuristics on Khan s 13 Available Test Problems Percentage Relative Error by CCFT and Other Heuristics on the Khan s 13 Available Test Problems Solution Quality for CCFT on 24 Additional Test Problems Performance Summary of the CCFT with Other Heuristics on 24 Additional Test Problems xi

14 7.22 Performance Summary of the CCFT with Other Heuristics on New MMKP Test Sets Percentage Relative Error of CCFT and Other Heuristics for the New MMKP Test Sets Computational Time in milliseconds by CCFT and Other Heuristics for the New MMKP Test Sets Solution Quality Comparisons Among All Heuristics on Khan s 13 Available Test Problems Performance Summary of All Heuristics on Khan s 13 Available Test Problems Percentage Relative Error Achieved by All Heuristics on Khan s 13 Available Test Problems Solution Quality of All Heuristics on 24 Additional Test Problems Performance Summary of All Heuristics on 24 Additional Test Problems Performance Summary of All Heuristics on New MMKP Test Sets Percentage Relative Error Achieved by All Heuristics on the New MMKP Test Sets Computational Time in milliseconds of All Heuristics on the New MMKP Test Sets Comparison of All heuristics based on Percentage of Optimum on the Khan s MMKP Test Problems Comparison of All Heuristics based on Percentage of Optimum on the New MMKP Test Sets A.1 New MMKP Test Problem Sets A.2 Exact Solutions for the New MMKP Test Problem Sets A.3 Exact Solutions for the New MMKP Test Problem Sets (Continued) B.1 Problem Set Parameters for Cho (2005) Competitive MDKP Test Problem Sets xii

15 Acknowledgement I would like to express my deep sense of gratitude to Dr. Raymond R. Hill for his exceptional guidance and support. He has been a great source of inspiration and strength throughout the course of my graduate studies, master s thesis research, and this dissertation work. His dedication and patience are the virtues which I will cherish for a long time and would strive to achieve them in my career. I am indebted to his encouragement and support in my research efforts from the beginning to the end. It was a great pleasure and honor to work under him. I would like to thank my dissertation committee members, Dr. Moore, Dr. Zhang, Dr. Kinney, and Dr. Rizki for taking time to serve on my committee and provide valuable suggestions for my work. Finally, I extend my great thanks to my family and close ones. I am grateful to my father, mother, Jatin, and Amita for their support, kindness, love and affection, which kept me going in achieving my career goals. I am deeply indebted to my parents for their active interest and encouragement in my education. My sincere thanks to Jatin for all the motivation, encouragement, and support throughout my studies and research at graduate school. xiii

16 Dedicated to Mummy, Daddy, Jatin, and Amita xiv

17 1. Introduction 1.1 Discussion of Knapsack Problems Heuristic techniques have long been used to quickly solve many combinatorial optimization problems. Finding an exact solution for an optimization problem in real practice is sometimes less practical in comparison to using an easily computed method of acquiring near-optimal solutions. When the problems grow larger in size, obtaining the exact solutions can take excessive computational time and storage space. In such cases, the results obtained by a complex, time consuming method may be no more attractive than near optimal solutions. Further considering the imprecision of the real-world problem data, and the approximate nature of some formulations, obtaining a precise solution in reality may seem meaningless. Obtaining a near-optimal solution in a reasonable computational time may be beneficial and more practical (Cho 2002). Heuristic optimization algorithms seek good feasible solutions to optimization problems where the complexity of the problem or the limited time available for its solution does not practically allow finding a guaranteed exact solution. Unlike exact algorithms, where time-efficiency is the main measure of success, two important issues that arise while evaluating heuristics are: how fast can solutions be obtained and how close are the solutions to optimal (Rardin and Uzsoy 2001). The binary knapsack problem (KP) is a non-trivial integer programming problem comprising binary variables, a single constraint with positive coefficients and binary restrictions on the variables. The KP is a difficult class of problems (Kellerer et al. 2004). The KP has been used to 1

18 model various decision making processes and finds a variety of real world applications: processor allocation in distributed computing systems, cutting stock, capital budgeting, project selection, cargo loading, and resource allocation problems. Industrial applications find the need for satisfying additional constraints such as urgency of requests, priority and time windows of the requests, and packaging with different weight and volume requirements. These necessities lead to the variants and extensions of knapsack problems. Multi-Dimensional Knapsack Problems (MDKP), Multiple Knapsack Problems (MKP), Multiple Choice Knapsack Problems (MCKP), and Multiple-choice Multidimensional Knapsack Problems (MMKP) are a few of the KP variants. The MDKP consists of selecting a subset of n given objects (or items) such that the total profit achieved by the selected objects is maximized while a set of knapsack constraints are satisfied. The MKP, MCKP, and MMKP involve the notion of sets or classes of items under consideration. These problems also find practical applications and form a more difficult set of KP problem types. Intratheater airlift problem, budgeting problem, adaptive multimedia problem, logistic scheduling problem, network design problem, warehouse container storage problem, resource allocation problem for satellites, airlift loading problem, and multiple container packing problem are applications where KP variants play an important role. Many heuristic methods have been applied to the KP and its variants providing solutions close to the true optimal. 1.2 Overview of the Dissertation Research Heuristic algorithms have been developed by many researchers to solve the knapsack problem and its variants and these have been tested on different problem sets. Not all the algorithms used to solve a given problem to optimality or near-optimality are equivalent in their performance. These performance differences could either be computational time, memory requirements, the solution quality, or the complexity of the algorithms. Some heuristic algorithms outperform other heuristic algorithms on certain types of test problems while they do not perform well on other test problems. Little research has examined why certain algorithms perform well on certain test problems while 2

19 not so well on other test problems. There has been little work done to gain knowledge of the test problem characteristics and their effects on algorithm performance with an objective of generalizing these performance insights into improved heuristics for practical problems. The organization of this dissertation is as follows. Chapter 2 briefly introduces the MDKP and discusses various approaches to solve the problem. This chapter provides background about branchand-bound, dynamic programming, greedy heuristics, transformation heuristics, and metaheuristic approaches to solve the MDKP. Chapter 3 discusses and provides a background about the extensions and variants of the knapsack problems: the MKP, MCKP, and MMKP. This chapter also discusses the formulations and the heuristic approaches for practical applications of the variants of the KP. Chapter 4 discusses the heuristic procedures and an analysis of existing test problems used for the variants of the knapsack problems. Chapter 5 discusses problem generation methods for the MMKP, develops a test problem generation method for the MMKP, and provides an empirical analyses of the heuristic solution procedures for the MMKP. Chapter 6 presents new greedy heuristics developed for the MMKP. Chapter 7 presents the metaheuristic approaches based on tabu search for solving the MMKP. Chapter 8 summarizes the research study, outlines the contributions, and identifies areas for further research. 1.3 Contributions of the Dissertation Research This research will focus on the MMKP, a complex variant of the knapsack problem. MMKP problems are complex and this complexity increases with the number of variables and the type of constraints, even to the point where constraints could make the problem non-linear. For example, in the case of a warehouse storage problem, the number of items to be stored, the space requirement of each item, and any incompatibility between items makes the problem complex and possibly nonlinear. For a real time airlift problem, when adding additional constraints that vary according to aircraft, like 3-dimensional aspects, aircraft center of gravity, aircraft floor loading considerations, and hazardous cargo restrictions, the problem can quickly become hard to solve. The increasing 3

20 complexity of such problems applied to the airlift logistics problem motivates this research to consider the variants of the knapsack problem. The contributions of this research are fourfold. The first contribution is to show how empirical science can lead to theory (Hooker 1994), as also evidenced in Cho (2005). The research involves collecting, coding, and testing the existing heuristics for the MMKP. This empirical analysis of the current heuristics will lead to new problem search space knowledge and the development of new heuristics. Past research has generally failed to study the reason why certain heuristic procedures outperform other heuristic procedures as a function of the test problem structure. The second contribution of the research study is to identify the performance traits of heuristic procedures and develop a more diverse set of MMKP test problems considering important problem characteristics like the number of variables, number of constraints, constraint correlation structure, and constraint slackness levels. The third contribution is the development of new greedy heuristic approaches for solving the MMKP. This development involves examining the existing heuristics performance against a new problem test set and using the subsequent analyses to develop new heuristic approaches. The fourth contribution is to develop improved metaheuristic solution procedures for the MMKP using the improved greedy heuristic approaches to initialize searches or to improve local search neighborhoods. 4

21 2. Multi-Dimensional Knapsack Problems 2.1 Introduction Suppose a subset of n items are to be packed into a knapsack of capacity c. Each item j has a profit p j and a weight w j. The 0-1 knapsack problem (KP) maximizes the total profit of the selected items without exceeding the knapsack capacity c. The 0-1 KP is formulated as: Maximize, subject to, n Z = p j x j (2.1) j=1 n w j x j c, (2.2) j=1 x j {0, 1}, j = 1,..., n (2.3) Equation (2.1) provides the total profit of selecting item j and Equation (2.2) ensures the knapsack constraint is satisfied. Equation (2.3) is the binary selection requirement on each decision variable x j such that x j = 1 if selected and x j = 0 if not selected. The 0-1 KP problem may have a collection of different resource constraints. Such a problem is called the Multidimensional Knapsack Problem (MDKP). A set of n items are to be packed simultaneously in m knapsacks with capacities c i for i = 1,..., m. Each item j has a profit p j and 5

22 weight w ij associated with knapsack i. The objective of the problem is to maximize the total profit of the selected items. The MDKP problem is formulated as: Maximize, subject to, n Z = p j x j (2.4) j=1 n w ij x j c i, i = 1,..., m (2.5) j=1 x j {0, 1}, j = 1,..., n (2.6) Equation (2.4) calculates the total profit of selecting item j and Equation (2.5) ensures each knapsack constraint is satisfied. Equation (2.6) is the binary selection requirement on decision variable x j such that x j = 1 if selected and x j = 0 if not selected. The value S i = c i / n j=1 w ij is called the slackness ratio of a constraint. Small values of S i mean constraints are tight; fewer items can fit in the knapsack. Large values of S i mean constraints are loose; more items can fit in the knapsack. Some real world applications of the MDKP are the cutting stock, capital budgeting, project selection, cargo loading, and resource allocation problems. Since MDKP is NP-hard (Frieze and Clarke 1984), the computation time increases rapidly as the problem size increases. As a result, a wide variety of heuristics have been applied to the MDKP. The remainder of this chapter briefly explains some of these solution approaches: Branch-and-Bound, Dynamic Programming, Greedy Heuristics, Transformation Heuristics, and Metaheuristic approaches. 2.2 Branch-and-Bound Approach The branch-and-bound (B & B) approach is an exact and potentially exhaustive enumerative approach used to determine an optimal solution for integer programming problems. The B & B approach is based on a complete enumeration of the solution space using specifically generated 6

23 constraints added to a problem to prune off regions of the solution space. In many cases, only a small subset of the feasible solutions are explicitly enumerated. Any solution subspace which is determined to not contain the optimal solution is excluded and thus its feasible solutions are noted as enumerated implicitly. We say the excluded subspace is pruned. The B & B algorithm is based on two basic principles: branching and bounding. Consider any problem with a finite solution space. In the branching part, a given subset of the solution space is decomposed into smaller subsets by including additional constraints. The union of these subsets must result in the complete solution space. The process of decomposition is repeated until in the limit, each solution subset contains a single feasible solution. The global optimal solution to the given problem is the best of all considered solutions. In a worst-case instance, every potential solution is considered. This process is represented by an enumeration tree. The leaves of the tree correspond to the points examined in a complete enumeration. Complete enumeration is not practical for problems where the number of variables, or nodes, in an integer program exceeds 20 or 30. Bounding plays an important role by intelligently avoiding complete enumeration. This part of the B & B algorithm derives lower and upper bounds for a given subset of the solution space. The lower bound can be chosen as the best integer solution encountered so far in the search process. In case of no solutions being considered, the objective value of a heuristic solution can be used as a lower bound. An upper bound is used to prune parts of the search space. Pruning by optimality, pruning by bound, and pruning by infeasiblity are the three ways to prune the tree and thus enumerate a large number of solutions implicitly (Wolsey 1998). In the following maximization example, suppose N = (U, L) = (UpperBound, LowerBound) is associated with a node in a B & B enumeration tree. Pruning by optimality: Suppose node N = (27, 13) is decomposed into two nodes N 1 = (20, 20) and N 2 = (25, 15). The values in the parantheses indicate upper and lower bounds on the nodes. Since the upper and the lower bound values on N 1 are equal, there is no further need to explore N 1. Hence, the rest of branch N 1 of the enumeration tree can be 7

24 pruned by optimality; the solution at N 1 is the best for that subspace including any subsequent subspaces. Pruning by bound: Suppose node N = (27, 13) is decomposed into two nodes N 1 = (20, 18) and N 2 = (26, 21). For N 2, the optimal value is at least 21 and the upper bound on N 1 is 20. No optimal solution can lie in N 1 because the solution in N 1 cannot exceed those in N 2. Thus, branch N 1 of the enumeration tree is pruned by bound. Pruning by infeasibility: Suppose node N = with upper bound 40 is decomposed into two nodes N 1 = (24, 13) and N 2. However, N 2 has no feasible solution in the subspace. The N 2 branch can be pruned by infeasibility. The B & B algorithm is a recursive procedure using tree-search facilities. This recursive structure employs different search techniques like the depth-first search or best-first search. The potentially exponential growth in computations as problem size increases makes the B & B approach less practical when the number of feasible solutions examined is very large unless early termination heuristics are used. Pierce (1968) used combinatorial programming based on two problem-solving procedures: use of a controlled enumerative procedure for considering all potential solutions and the elimination from explicit consideration of particular potential solutions due to dominance, bounding and feasibility considerations. Pierce observed that the search directed first to the discovery of a feasible solution and then to successively better feasible solutions until an optimal solution is determined. Thesen (1975) developed a B & B algorithm for the MDKP. He defined a recursive tree structure for B & B with the objective of reducing computer storage requirements. Thesen s algorithm was tested against algorithms by Geoffrion (1967) and Zionts (1972). The test results indicated that Thesen s algorithm took one tenth of the solution time of the other two algorithms. Results also indicate solution times increased for all three algorithms with tighter problem constraints (lower S i values). The magnitude and the rate of increase in solution times with the number of variables 8

25 was less for Thesen s algorithm in comparison with the other two algorithms. Thesen s algorithm showed improved execution times for larger test problems but could not solve problems with more than 50 variables. Shih (1979) proposed a modified B & B approach to solve the MDKP. A solution was found by an LP relaxation of the KP, obtained using each of the m constraints and then using the minimum value among the resulting m objective function values as a bound for each node. Shih tested his algorithm on randomly generated problems with 5 constraints and variables. Shih s proposed algorithm measured 13 minutes versus 380 minutes taken by a Balas approach. Balas and Zemel (1980) developed an algorithm for KP based on three concepts. The first concept focuses on the core problem which is based on the knowledge of an optimal solution. Suppose the items are ordered in a decreasing order of their cost/weight (p j /w j ) ratios and let an optimal solution vector be given by x. Define a := min{j x j = 0} and b := max{j x j = 1} a 1 a 1 in an optimal solution. If p = p j and w = w j, then the core is represented by the items in j=1 j=1 the interval C = {a,..., b} and the core problem is defined as Maximize, Z = j C p j x j + p (2.7) subject to, w j x j c w, (2.8) j C x j {0, 1}, j C (2.9) The core problem is the subproblem in those variables whose cost/weight ratio falls between the maximum and the minimum c j /w j ratio for which x j has a different value in an optimal solution to the KP from that in an optimal solution to the LP relaxation (LKP). The size of the core is a small fraction of the entire problem size and does not increase rapidly with problem size (Pirkul 1987). Identifying the core requires solving the KP. Some authors use an approximation in solving the LKP ((Pisinger 1999a) and (Pirkul 1987)). A second concept focuses on the binary-search type method 9

26 to solve the LKP without sorting the variables. A third concept focuses on the development of a heuristic to solve the KP to optimum. This heuristic approach uses implicit enumeration to find the optimal solution of the approximated subproblem. Other research, discussed later, refers back to this core problem concept. Martello and Toth (2003) used greedy heuristic algorithms based on the solutions to the Lagrangian and surrogate relaxations, and a reduction approach to determine the optimal value of a high percentage of variables. These results were then used in a B & B algorithm to find the exact solution of 2KP. 2.3 Dynamic Programming Dynamic programming is an exact and exhaustive enumerative approach that can find the optimal solution for MDKP by solving a small subproblem and then extending this solution iteratively until the complete problem is solved. Assuming that the optimal solution of the knapsack problem was computed for a subset of the items and all capacities c, add an item to this subset and check whether or not the optimal solution needs to be changed for this enlarged subset. This can be found by using the solutions of the knapsack problem with smaller capacity and computing the possible change of the optimal solutions for all capacities. This procedure of adding an item to the subset is repeated until all items have been considered and the overall solution is obtained. Kellerer et al. (2004) explain the concept of Dynamic Programming for KPs as follows. Consider the subproblem of a KP with item set {1,..., j} and knapsack capacity d c. The knapsack problem for {j = 1,..., n} and {d = 0,..., c} is defined as: Maximize, Z j 1 (d) = j p l x l (2.10) l=1 10

27 subject to, j w l x l d, (2.11) l=1 x l {0, 1}, l = 1,..., j (2.12) with optimal solution Z j 1 (d) and binary decision variable x l such that x l = 1 if selected and x l = 0 if not selected. If the optimal solution value Z j 1 (d) for all capacity d = {0,..., c} is known, then consider an additional item j added to the subset of items. The corresponding optimal solution value Z j (d) can be found using the recursive formula: Z j (d) = { Z j 1 (d) if d < w j, max{z j 1 (d), Z j 1 (d w j ) + p j } if d w j (2.13) Weingartner and Ness (1967) developed algorithms using the dynamic programming framework to solve MDKP. They introduced a dual approach in dynamic programming which starts with all the items included in a knapsack. Their approach systematically removes items from the knapsack until feasibility is obtained. The size of the problem was reduced by heuristically determining lower and upper bounds. The lower bound elimination method is used to determine the lower bound. This method eliminates any solution strategy at any stage from further consideration that, if it were augmented with all the items remaining to be considered, would yield a solution less than the best solution already achieved. This method requires finding the best feasible solution. The upper bound is calculated by a linear programming method. Soyster et al. (1978) developed an algorithm with an implicit enumeration strategy applied to the MDKP subproblems. The subproblems were generated by the LP-relaxation of the MDKP. They solved the LP-relaxation of the MDKP to optimality and partitioned the optimal solution into fractional and integer components. They applied implicit enumeration to solve the subproblems with fractional valued variables obtained in the linear program (LP). At each subsequent iteration, the size of the subproblem was increased by adding a variable. By applying implicit enumeration to subsequent subproblems, they achieved a non-increasing sequence of optimal values of the LP 11

28 which served as the upper bound for the IP. The algorithm terminated when the difference between the current solution and the current upper bound was less than one. The Soyster et al. s (1978) algorithm follows the dynamic programming approach of adding a variable to the subproblem and solving it until it meets the termination criterion. Pisinger (1997) presented a minimal algorithm for the knapsack problem where the enumerated core size is minimal. The minimal algorithm was based on a breadth-first, dynamic programming approach for the enumeration as the core problem is gradually extended. The enumerated core problem is the smallest possible symmetrical core problem which is enumeratively solvable. Pisinger uses stronger upper bounds for reducing the items that are not present in the core. This algorithm gave stable solution times in computational experiments conducted on randomly generated uncorrelated, weakly correlated, strongly correlated, and subset-sum data instances (these types of test problems are discussed in later chapters). Martello et al. (1999) developed an algorithm based on the dynamic programming approach by Pisinger (1997) using the cardinality bounds approach by Martello and Toth (1997) to generate valid inequalities. They used surrogate relaxation to solve the problem to optimality. The new algorithms reduced the computation time and memory space requirements compared to previous efforts. Bertsimas and Demir (2002) presented the Approximate Dynamic Programming (ADP) approach for the MDKP. They approximate the optimal value function by using a parametric method (ADP-P), a nonparametric method (ADP-N), and a base-heuristic (ADP-BH). In the dynamic programming formulation, the optimal value function is obtained by taking the maximum of the preceding objective values of the subproblem. The ADP-BH estimates the optimal value function by the solution value of a suboptimal methodology to the corresponding subproblem. This suboptimal methodology is the base-heuristic approach. The ADP-P and ADP-N algorithms are based on the research results from the probabilistic analysis of the MDKP in Frieze and Clarke (1984). The computational results showed that their ADP approach using the new heuristic was fast and accurate compared to the ADP approach based on parametric and nonparametric methods. 12

29 2.4 Greedy Heuristics Heuristic algorithms are designed to find good, but not necessarily optimal solutions, quickly. A very simple greedy approach for the knapsack problem can be obtained by considering the profit to weight ratio (p j /w j ) of every item and trying to select items with the highest ratio for the knapsack. This approach seeks to generate highest profit while consuming little knapsack capacity. Consider the KP of equations (2.1) to (2.3). The greedy heuristic considers arranging the items in a decreasing order of profit to weight ratios (p 1 /w 1 p 2 /w 2... p n /w n ). A greedy algorithm begins with an empty knapsack, considers the items arranged in decreasing profit to weight ratios, and adds every item under consideration into the knapsack until the capacity constraint denoted by Equation (2.2) is violated. Cabot (1970) used a combination of enumeration and elimination techniques for solving linear equalities. The enumeration method is applied to the MDKP by treating each constraint as one would in a one-dimensional knapsack problem and then generating bounds by looking at the intersection of the bounds generated by each set of constraints. This technique was tested on 50 randomly generated problems with ten variables and five constraints to determine the effect that the order of the variables had on the solution time. The results were quite poor in terms of the computation time, as the bounds generated were poor. Senju and Toyoda (1968) present an approach for finding approximate solutions for the MDKP using dual methods. Starting with all items selected, they used a Dual Effective Gradient Method to evaluate the utility of every element and deselect items until feasibility was obtained. The Dual Effective Gradient Method starts with an infeasible solution and proceeds to search for a feasible solution. Toyoda (1975) introduced the Primal Effective Gradient Method to obtain approximate solutions to MDKP programming problems. The Primal Effective Gradient Method starts with a feasible solution (nothing selected) and then attempts to improve upon this solution. It assigns measures of preferability to each decision variable. This measure corresponds to the profit per unit of aggregate necessary resource known as the pseudo-utility ratio. The pseudo-utility ratio is the 13

30 effective gradient given as (p j /(uw j )) where u is a multiplier for a column vector x j that weighs the resource cost per constraint against the level of constraint utilization. The most preferred variables are set from zero to one. This method gives very good approximate solutions for MDKP problems. Loulou and Michaelides (1979) developed four different heuristic methods by varying an index of aggregate consumption of resources by each x j also known as the penalty vector which is used for calculating the pseudo-utility ratio. These heuristic methods were used to obtain approximate solutions to MDKP. They tested their algorithms on various problems of different sizes. Their analyses showed that their methods yielded better results than Toyoda s approach (upon which their four methods were based). The average relative errors were less than 1% of the optimal solution to each of the test problems used. Balas and Martin (1980) devised the pivot and complement method to solve 0-1 problems. The procedure first solves the linear program relaxation and then performs a sequence of pivots aimed at exchanging non-basic slack variables for a basic 0-1 variable at a minimal cost. Finally, a local search based on complementing certain sets of 0-1 variables tries to further improve the solutions. The experimental results carried out on 92 test problems showed that the time taken to run the linear program was very high. Optimal results were obtained for 49 of the problems. Lee and Guignard (1988) developed and tested a heuristic for MDKP. Their heuristic involves two phases. Phase I finds a feasible solution by applying a modified Toyoda (1975) heuristic and reduces the problem size by fixing variables using LP-relaxation. The modified Toyoda s method consists of selecting as many variables as possible each iteration until the solution obtained is feasible (versus selecting only one variable each iteration as in the original Toyoda s heuristic). Phase II of the heuristic approach improves the solution obtained in Phase I by complementing certain sets of variables using the Balas and Martin (1980) method. On comparing the Lee and Guignard (1988) algorithm with Toyoda (1975) and Magazine and Oguz (1984), Lee and Guignard s algorithm yielded better solutions in less computation time while the Balas and Martin (1980) approach showed better performance over the Lee and Guignard (1988) approach in terms of solution quality. 14

31 Vasquez and Hao (2001) presented a hybrid approach for the MDKP; a combination of linear programming and tabu search. The linear program is used to obtain promising continuous optima and then tabu search is used to carefully and efficiently explore binary areas close to these continuous optima. They considered several additional constraints such as the hyperplane constraint, geometrical constraint, and qualitative constraint to solve the MDKP problems. This algorithm showed significant improvements when tested on more than 100 benchmark instances. Cho et al. (2003b) conducted an empirical analysis of three legacy heuristic methods: Toyoda (1975), Senju and Toyoda (1968), and Loulou and Michaelides (1979) using Hill and Reilly s two constraint MDKP test set (Hill and Reilly 2000), a robust problem set in terms of varied constraint slackness and correlation structure. The empirical analysis led to finding a best performer among the heuristics for particular problem types. A new type-based heuristic was developed based on problem pre-processing and selecting a preferred heuristic, the first approach of its kind for MDKP. Computational results indicated the type-based heuristic showed slightly better and more consistent performance in comparison to the three different heuristic methods for varied test problems. Cho et al. (2003a) developed a heuristic using Lagrange multipliers to transform a MDKP into an unconstrained problem, categorizing as selected, unselected or uncertain variables, and then solves the remaining core problem using a greedy heuristic. The greedy heuristic used is the Toyoda (1975) heuristic. The heuristic developed accounts for important problem characteristics such as constraint slackness and correlation structure. The algorithm is tested on Hill and Reilly s test set (Hill and Reilly 2000). The results show the new heuristic has better performance at all constraint slackness levels when compared to a suite of competitor heuristics. Cho et al. (2006) developed a new primal effective gradient heuristic. The new gradient function was obtained by combining the characteristics of heuristics by Senju and Toyoda (1968) and Loulou and Michaelides (1979). Subsequent testing using Hill and Reilly s test set (Hill and Reilly 2000) demonstrated improved performance 69% of the time compared to the results obtained by the legacy approaches. Further, the new approach dominated the legacy heuristic approaches when 15

32 applied to modified versions of benchmark test problems and to problems specifically created to provide harder problem test instances while mimicking the size of current benchmark problems. 2.5 Transformation Heuristics KP is a MDKP with m = 1. KP is well studied and there exist more exact algorithms and heuristics for KP than MDKP (Cho 2005). A transformation heuristic with relaxation techniques like Fisher s Lagrangian relaxation, Glover s Surrogate relaxation, and composite relaxation can reduce the MDKP to a KP approximation of the MDKP. Solutions to the KP approximation give heuristic solutions to the original problem. Consider the MDKP formulation equations (2.4) to (2.6) with m 1 additional constraints: n w kj x j c k, k = m + 1,..., m + m 1 (2.14) j=1 Define L(λ) as the Lagrangian relaxation of the MDKP, Equations (2.4) through (2.6) and (2.14). Let λ = (λ 1, λ 2,..., λ m ) be a vector of non-negative Lagrangian multipliers. A Lagrangian formulation moves difficult constraints into the objective function. We assume without loss of generality that constraints (2.5) are difficult. The Lagrangian relaxation of the MDKP, L(λ), becomes: Maximize, subject to, n m n Z(L(λ)) = p j x j λ i ( w ij x j c i ) (2.15) j=1 i=1 j=1 n w kj x j c k, k = m + 1,..., m + m 1 (2.16) j=1 x j {0, 1}, j = 1,..., n (2.17) λ i 0, i = 1,..., m (2.18) Note constraint (2.5) is in the objective function as the penalty term: 16

33 m n λ i ( w ij x j c i ) (2.19) i=1 j=1 Violations of constraint (2.5) results in the penalty term (2.19) reducing the objective function. The objective function value of the original MDKP is not larger than the objective function value of the relaxed problem (Fisher 1981). Thus, for a vector of non-negative Lagrangian multipliers, the optimal solution to the relaxed problem is an upper bound on the original problem (Fisher 1981). A surrogate relaxation replaces some set of constraints with a linear combination of those constraints. Glover (1968) discusses surrogate bounds and Glover (1977) uses a surrogate-based heuristic to solve the MDKP. Let S(µ) denote the surrogate relaxation of the MDKP problem represented by Equations (2.4) through (2.6). Assume that the m capacity constraints are difficult to solve. These constraints can be merged into a single constraint instead of removing them from the problem. Let µ = (µ 1, µ 2,..., µ m ) represent a vector of non-negative multipliers. The surrogate relaxed problem is given as: Maximize, subject to, i=1 j=1 i=1 n Z(S(µ)) = p j x j (2.20) j=1 m n m µ i ( w ij x j ) µ i c i, (2.21) x j {0, 1}, j = 1,..., n (2.22) µ i 0, i = 1,..., m (2.23) All feasible solutions to the original MDKP problem are also feasible for the surrogate relaxed problem. For all the non-negative multipliers µ, the optimal solution for the surrogate relaxed problem is an upper bound to the original problem (Glover 1977). Composite relaxation combines Lagrangian relaxation and surrogate relaxation. The goal is to make computations easier while the Lagrangian relaxation penalty keeps the solution close to 17

34 the feasible region (Kellerer et al. 2004). Consider the MDKP problem of Equations (2.4) through (2.6). Let Co(λ, µ) denote the composite relaxation, formulated as: Maximize, subject to, n m m Z(Co(λ, µ)) = (p j λ i w ij )x j + λ i c i (2.24) i=1 j=1 i=1 j=1 i=1 i=1 m n m µ i ( w ij x j ) µ i c i, (2.25) x j {0, 1}, j = 1,..., n (2.26) Gavish and Pirkul (1985) tested and compared Lagrangian, surrogate, and composite relaxation bounds for the MDKP. Computational tests indicated that composite relaxation provided the best bounds. The authors then suggest rules for reducing the problem size. They solve the reduced problem by a modified B & B approach. They compared their algorithm performance to Shih (1979) and Loulou and Michaelides (1979) based on computing times and solution quality. They found that their algorithm improved solution quality with shorter computing times. Osorio and Hernandez (2004) generated new MDKP constraints by using constraint pairing and initial integer solution to combine the dual surrogate constraint with the objective function. These constraints were used for generating different types of logic cuts which were included in the B & B framework to solve the problem to optimality. The developed approaches were tested on smaller and larger sets of MDKP test problem instances developed by Weingartner and Ness (1967), Senju and Toyoda (1968), Shih (1979), and Chu and Beasley (1998). The comparative analysis indicated that the logic cuts in the B & B model improved the solution performance reducing the number of nodes in the search tree. 2.6 Metaheuristic Approaches Metaheuristics are approximate algorithms that combine basic heuristic methods into higher level frameworks to efficiently and effectively explore the search space (Blum and Roli 2003). Meta- 18

35 heuristics, a term coined by Glover and Greenberg (1986), is derived from the two Greek words; heuristics which means to find and meta which means beyond. Various heuristic approaches mostly based on analogies with natural phenomena, like tabu search, genetic algorithms, and simulated annealing were developed in the later 1970s. Metaheuristics have been used to tackle many difficult combinatorial optimization problems. A good metaheuristic implementation is likely to solve a combinatorial optimization problem in a reasonable computation time (Gendreau and Potvin 2005) and empirical evidence suggests the solutions are generally of high quality Tabu Search (TS) Glover and Greenberg (1986) proposed tabu search (TS) as an extension of local search (LS) to allow LS to overcome local optima. TS is a metaheuristic procedure, which uses a local search subroutine in order to find local optima. The basic principle of TS is to guide LS when it comes across local optimum by allowing non-improving moves. This involves explicitly preventing the LS cycling back to previously visited solutions by using memory structures known as tabu lists. A basic TS combines LS with short term memory. TS has been applied to a wide range of optimization problems to include various classes of integer problems like planning, design, and scheduling. Search space and the neighborhood structure are two basic elements of a TS heuristic. The search space is the space of all possible solutions that can be visited during a search. TS is based on tabu structures used to prevent LS cycling among solutions when moving away from local optima regions via non-improving LS moves. TS prevents the search from retracing itself by tracking moves accomplished based on move attributes employed. If potential moves have been used, the move is declared tabu. These tabu moves force the search away from previously visited portions of search space, providing more extensive exploration of the search space. Tabu lists can be fairly simple, can be complex, or may use hashing techniques to maintain a complete search history of all solutions visited. The TS search process is controlled by the length of the tabu list known as tabu tenure. Smaller 19

36 tabu tenures make the search concentrate on smaller areas of the search space while larger tabu tenures force the search to explore larger areas. Robust algorithms can vary the tabu tenure during the search process. Tabu tenures can be dynamically changed; in the case of higher diversification, tabu tenure is increased if there are repetitive solutions, and in the case of intensification, tabu tenure is decreased if there are no improvements in the solutions (Blum and Roli 2003). Although a tabu structure can be a powerful search mechanism, it can prohibit attractive move attributes which may in turn lead to a stagnation in the search process. Thus, aspiration criteria for tabu structures allow the TS to override tabu status. The simplest and the most commonly used aspiration criteria is to allow a tabu move if it results in a new best known solution (Glover and Kochenberger 2002). Tabu list structures are called short-term memory structures but are often augmented with other memory structures. Various data pertaining to the search are compiled to create memory. These data are then used to make strategic changes to the TS process. Intermediate-term memory is used to intensify a tabu search in promising areas. The most common form of intermediate memory is the elite list. The elite list is simply a list of good solutions compiled during the TS process. These good solutions may in fact be good solutions evaluated by the search process but not selected in the TS move. Intermediate-term functions involve restarting the TS from solutions stored in the elite list. Recency, frequency, quality, and influence data are examples of long term memories. Receny based memory records the most recent iteration for every solution. Frequency based memory records the number of times a solution has been visited. These data provide information about whether or not the search is confined to a specific region. Quality based memory gathers and extracts information of the search history to identify good solution components. Influence based memory has information about the choices made during the search process and indicate the choices that have been critical (Blum and Roli 2003). While the short-term and intermediate-term memory components of TS intensify the search, 20

37 long-term memory structures are used to diversify the search into new areas of the search space. A key development of an effective TS approach involves the implementation of rules or triggers that exploit memory to systematically intensify and diversify the search. A tabu search approach for solving zero-one mixed integer programming problems exploiting the extreme point property of zero-one solutions is due to Løkketangen and Glover (1998). In the first part of their research, a basic first level TS is explored. The second part involves the extension of the first-level TS mechanisms to include developing diversification schemes for driving the search process into more fruitful regions of the search space. The study involved probabilistic moves and the study of search processes to identify new decision rules for diversification to produce good results. The experimental tests were carried out on 57 MDKP benchmark problems. Optimal results were found for 54 of the problems. Hanafi and Freville (1998) devised a TS based on strategic oscillation (SO) and surrogate constraint information that provides a balance between intensification and diversification strategies. Strategic oscillation is a TS concept involving moves across boundaries that normally do not admit moves, for example crossing the feasible-infeasible boundary. The SO critical solutions considered by Hanafi and Freville are solutions lying on or anywhere near the boundary of the feasible domain. These critical solutions form what they call the promising zone. The information deduced from the surrogate constraints and memory of the search controls the crossing and intensive exploration of this promising zone. A forward path from the feasible domain to the infeasible region and return paths are helpful in achieving the crossing exploration. These paths are built by performing constructive and destructive phases on the solution. A constructive phase (ADD) adds items to the knapsacks causing a solution trajectory into the infeasible region. A destructive phase (DROP) removes items until solution feasibility is restored. An intensification strategy is activated each time a promising zone is reached. The experimental tests were carried out using two sets of MDKP instances. The first set from Fréville and Plateau (1986) involved 54 instances and the second set from Glover and Kochenberger (2002) involved 24 instances. Optimal solutions were obtained for each of the instances and the previously best known solutions were improved for five of the last seven 21

38 instances. Aboudi and Jornsten (1994) applied tabu search for solving MDKP using the Pivot and Complement Heuristic by Balas and Martin (1980) as implemented in ZOOM/XMP. The experimental results were tested on 57 standard MDKP problems; optimal solutions were found for 49 out of the 57 problems (86%) Genetic Algorithm (GA) Genetic algorithms are search procedures inspired by biology and the workings of natural selection. Conceived by John Holland and his colleagues, GAs have been applied in many diverse applications (Chu and Beasley 1998). The GA name originates from the analogy between the representation of a complex structure by means of a vector of components, and the idea, familiar to biologists, of the genetic structure of a chromosome (Reeves 1993). In biology, natural selection reinforces characteristics most amenable to a species survival. Genes within the chromosomes of the stronger members, corresponding to the more desirable characteristics, pass to subsequent generations through a reproduction process. In a GA generation, genetic operators are applied to selected individuals (solutions) from the current population (of solutions) to create a new population. Three main genetic operators: reproduction, crossover, and mutation are generally employed. Varying probabilities of applying these operators can control the speed of convergence of the GA. The choice of crossover and mutation operators are most important to the performance of the GA. Hence, crossover and mutation operators must be carefully designed. (i) Reproduction: Reproduction is a process in which a new population is created by copying individual members, or portions of members, from the present population as a function of their fitness function values (Goldberg 1989). This fitness function could be some measure of profit, utility or goodness that we need to maximize. Copying members according to their 22

39 fitness values means that member solutions with higher value have a higher probability of contributing to one or more offspring in the next generation. This gives the possibility of survival for good solutions and an element of immortality to the problem solution or population member. (ii) Crossover: New individuals are created as offspring of two parents via the crossover operation. One or more crossover points are selected at random within the chromosome of each parent. The parts that are delimited by the crossover points are then interchanged between the parents. The individuals formed as a result of such an interchange are the offspring (Zalzala and Fleming 1997). Burst crossover involves crossover at every bit position, thus possibly resulting in many crossover-points, depending on the selected probability (Hoff et al. 1996). Common types of crossover are depicted in Figure 2.1 ((Zalzala and Fleming 1997) and (Renner and Ekart 2003)). It is better to keep strong members intact in the later phases of a GA. Hence, it is a good idea to use an adaptively changing crossover rate (higher rates in the early phases and a lower rate at the end of a GA). In some cases it is also desirable to use several different types of crossover at different stages of evolution as shown in Figure 2.1 ((Zalzala and Fleming 1997) and (Renner and Ekart 2003)). (iii) Mutation: A new individual is created by modifying a selected (existing) individual ((Zalzala and Fleming 1997) and (Mitchell 1996)). These modifications may be in the form of changing one or more values in the representation or in adding/deleting parts of the representation. In GAs, mutation is a source of population diversity and is applied in addition to crossover and reproduction. Mutation is considered a background operator. Mutation provides a guarantee that the probability of obtaining any solution is not zero and acts as a means to recover good genetic material which may be lost through crossover. In GAs, mutation is randomly applied with low probability, typically in the range of to 0.01 per bit, and modifies elements in the chromosomes (Zalzala and Fleming 1997). Different kinds of mutation op- 23

40 erators have been developed. Inverted mutation is mutation per bit, wherein each bit receives the opposite value of what it had previously (Hoff et al. 1996). For example, if the chromosome before mutation is , the chromosome after inverted mutation may be Other mutation operations include trade mutation (Lucasius and Kateman 1997), whereby the contribution of individual genes in a chromosome is used to direct mutation towards weaker terms, and reorder mutation (Lucasius and Kateman 1997), which swaps the positions of bits or genes to increase diversity in the decision variable space. Figure 2.1: Different types of crossover operators ((Zalzala and Fleming 1997) and (Renner and Ekart 2003)) A GA begins with a group of solutions called a population. Figure 2.2 (Renner and Ekart 2003) shows a general flowchart of a genetic algorithm. 24

41 Figure 2.2: Genetic Algorithm Flowchart (Renner and Ekart 2003) The GA works as follows: (i) The initial population is comprised of individuals, usually created randomly. The population obtained may vary in terms of proportion of population members feasible for the specific problem (some GA approaches guarantee feasibility of all members). (ii) A fitness measure is used to evaluate each individual in the current population. (iii) If the termination criterion is satisfied, the best solution or set of solutions, is returned. (iv) Based on the computed fitness values, individuals are selected from the current population. A new population is formed by applying genetic operators (reproduction, crossover, mutation) to these selected individuals. The selected individuals are called parents and the resulting 25

42 individuals are called offspring. Implementations of GAs differ in construction of each new population. Some implementations extend the current population by adding new individuals and then downsize to a new population by omitting the least fit individuals. Other implementations create a separate population of new individuals by applying genetic operators. (v) Actions starting from step (ii) are repeated until a termination criterion is satisfied. Each iteration is called a generation. The GA termination criterion is usually some predefined number of iterations or evidence that future solutions will not improve. The next generation is based on selecting parents and offspring for survival again according to some predefined strategy. Non-selected solutions die and are removed from the population. Hoff et al. (1996) showed how proper selection of parameters and search mechanisms yields a genetic algorithm producing high quality solutions for MDKP. They examined parameters based on recommendations from the literature, and conducted empirical tests to find a best set of parameters. They also used routines to ensure all solutions were feasible, called filter methods. The experimental tests were carried out on 57 standard test problems for MDKP. The optimum results were found for 56 of the 57 test cases. Chu and Beasley (1998) incorporated a heuristic operator using problem-specific knowledge into the standard genetic algorithm approach. Binary representation was used to represent the solution. Tournament selection was used to select the parents and uniform crossover and flip mutation were used to generate offspring. The population size of 100 was used with steady state replacement and two bits per child string mutation rate were used. An ADD/DROP heuristic approach was used in order to improve the fitness of a feasible solution and to obtain a feasible solution from an infeasible solution, respectively. The objective function value was used to calculate the fitness value. The ADD/DROP operator fixes infeasible chromosomes using Toyoda s concept of aggregate resource for knapsack problems. The GA algorithm performance was first tested on the standard test problems. The first set consisted of 55 problems, which were used in earlier research ((Aboudi and Jornsten 1994), (Løkketangen and Glover 1998), (Hoff et al. 1996), and (Løkketangen 1995)). 26

43 Optimal solutions were found for all 55 test problems. A second test set of 270 large problems was generated randomly. The results obtained indicated that the GA heuristic was very effective for these large problems. The GA was capable of converging to high quality solutions. These 270 problems have since become de facto standard problems for MDKP analyses. Thiel and Voss (1993) used four different techniques for solving MDKP problems. The first technique used a penalty function approach. They found that the degree to which the fitness of the infeasible solutions was penalized is important. If the penalty function is too restrictive, the GA converged to suboptimal solutions. If the penalty function is too loose, infeasible solutions dominate the population. The penalty function they implemented considered three cases and evaluated solutions according to the fitness value and the distance from feasibility. Case 1 was if the solution is feasible and the penalty function value was zero. If the infeasible chromosome value was less than the population mean, it was considered a very poor solution and a fitness value of 1.0 was assigned to it, effectively removing it. If the fitness value of the chromosome is above the average fitness value, then it is penalized by an infeasibility distance measure. A second technique implemented used the ADD/DROP operator. The ADD/DROP operator starts with dropping an item and in subsequent steps other items are added while preserving feasibility of the solution. Items are selected based on a pseudo-utility measure. Thiel and Voss s third technique prevented solutions from becoming infeasible. Their filter operator begins testing whether or not a certain solution is feasible. If a solution is infeasible, random items are removed until a feasible solution is obtained. Their fourth technique used a tabu operator. The tabu operator is applied to either a randomly determined chromosome string or a chromosome string with the highest fitness of a population. The tabu operator helps in avoiding local optimality and providing improved solutions within the search of the GA. This operator is only applied to feasible strings and hence the filter operator is processed before the tabu operator in order to make the chromosome string feasible. The results obtained by the penalty function were not satisfying. The ADD/DROP and the 27

44 filter operator improved the solution performance. An improvement by 0.5% in the solutions was obtained by using the filter and the DROP/ADD operator. The results obtained using the tabu operator proved the best. Optimal solutions were obtained for most of the test problems at the expense of increased computing time using the tabu operator. Hill and Hiremath (2005) developed and tested a new initial population generation approach using information about the problem structure to solve MDKP using GAs. This approach focused on generating initial populations that are stronger in solution quality, solution diversity, and hover near the boundary of feasible and infeasible solutions within the problem solution space. The new approach developed involved randomly setting more variables to one when the constraints are loose and setting fewer variables to one when the constraints are tight. Computational tests were performed on Hill and Reilly s (2000) test instances. Compared to a traditional random initial population generation approach, this new approach shifted the best so far curve to the left, thus reducing time to reach good solutions. This showed that a better initial population helps GA convergence. The improved performance was notable for the harder problems with tighter constraints Simulated Annealing (SA) Simulated annealing (SA) is a local search algorithm with a random element. A simple form of local search (a descent algorithm) starts with an initial solution, usually chosen at random. A neighbor of this solution is then generated by some suitable mechanism and the change in cost is calculated. If the cost is reduced, the generated neighbor replaces the current solution. If the cost is higher, the generated neighbor replaces the current solution with some probability. This probability changes over the course of the algorithm. A high probability causes the SA to function like a random search. A low probability causes the SA to function like a steepest descent algorithm. The probability varies based on a thermodynamic function involving a temperature parameter. The process is repeated until no further improvement can be found in the neighborhood of the current solution and the descent algorithm terminates at a local minimum (Eglese 1990). The efficiency of SA depends on 28

45 the neighborhood structure used. The key feature of SA is that it provides a means to escape local optima by allowing hill-climbing moves in hopes of finding a global optimum (Henderson et al. 2003). Drexl (1988) used a Probabilistic Exchange (PROEXC) to solve the MDKP. SA is a random local search, which allows non-improving moves with probability t called temperature. After r repetitions at temperature t, the temperature is reduced by a factor ϕ and the repetitions are increased by a factor ρ. The author initialized t as, t = α β (2.27) where, α = max{c j j } min{c j j } (2.28) The author found that PROEXC works best at β= 0.5 and ϕ= 0.6. It was found that n was a good value for r, and 1.2 was a good value for ρ. The experimental results of the PROEXC were tested on 57 standard problems with known optimal solutions. It was found that the PROEXC executed very fast giving good solutions. PROEXC solution quality was found to be superior to a deterministic variant, DETEXC, but the DETEXC was faster in terms of execution time. DETEXC is equivalent to PROEXC except that only changes improving the objective function value are performed, thus making DETEXC effectively a steepest ascent algorithm. 29

46 3. Extensions and Variants of the Knapsack Problem involving the notion of sets 3.1 Introduction Knapsack problems find applications in cargo loading, capital budgeting, cutting stock, and resource allocation. However, industrial applications must consider additional constraints such as time windows, priority requests, different weights and volumes. Such applications lead to many variations and extensions of the KP. Since these extensions of the basic knapsack model address practical optimization problems, these more general variants of the KP have become standard problems (Kellerer et al. 2004). Bounded Knapsack Problems, Multiple Knapsack Problems, Multiple-Choice Knapsack Problems, and Multiple-choice Multi-dimensional Knapsack Problems are the KP variants discussed in the following sections. Each of these KP variants involve the notion of sets or classes of items under consideration. 30

47 3.2 Multiple Knapsack Problems (MKP) MKP Formulation The Multiple Knapsack Problem is the generalization of the standard knapsack problem from a single knapsack to m knapsacks each with different capacities. Given n items, partition these into m knapsacks with capacities c i, i = {1,..., m}. Each item j has a profit p j and weight w j. The objective is to maximize the total profit of the selected items. The MKP is formulated as: Maximize, subject to, Z = m n p j x ij (3.1) i=1 j=1 n w j x ij c i, i = 1,..., m (3.2) j=1 m x ij 1, j = 1,..., n (3.3) i=1 x ij {0, 1}, i = 1,..., m, j = 1,..., n (3.4) Equation (3.1) provides the total profit of assigning item j to knapsack i (x ij = 1). Equation (3.2) ensures each knapsack constraint is satisfied while Equation (3.3) ensures each item is assigned to at most one knapsack. Finally, Equation (3.4) is the binary selection requirement on decision variables x ij such that x ij = 1 if selected and x ij = 0 if not selected Heuristic Solution Approaches for MKP Hung and Fisk (1978) developed and tested a B & B algorithm based on a depth-first search for solving the MKP. This procedure involved constructing higher levels of the decision tree either by assigning an item to a knapsack or removing an item from a knapsack. The authors used Lagrangian and surrogate relaxation techniques to reduce the MKP to a single knapsack problem. The item assigned at every level of the tree depends on the type of the relaxation. The algorithm with 31

48 Lagrangian relaxation selects an item that appears in the most single knapsack solutions while the algorithm with surrogate relaxation selects an unassigned item with the lowest-index (p j /w j ). The algorithm considers knapsacks with increasing capacities and the items are assigned to the knapsacks in an increasing order of the indices. Testing the algorithms with both types of relaxations showed that for a smaller number of items and boxes, the surrogate relaxation was relatively more efficient in terms of the number of problems reaching the optimum and in terms of computation time. For larger problems, the Lagrangian relaxation was better. On average the solution times for Lagrangian relaxation were higher than those for surrogate relaxation. Martello and Toth (1985) developed and tested an algorithm (MTM) using a branch-and-bound approach. Their algorithm exactly solves the problem using a depth-first tree-search. This algorithm was tested on randomly generated test problems. The computational results showed that the time required to find the exact solution increased more rapidly with number of knapsacks (m) than with the number of variables (n), and was impractical for m > 10. When changing the number of backtrackings to 10 and 50, the algorithm yielded solutions close to the optimum with reasonable computational times which increased slowly in both n and m. Pisinger (1999b) developed an exact algorithm (called MULKNAP) incorporating a branchand-bound approach to solve the MKP. His recursive branch-and-bound approach applied surrogate relaxation for deriving upper bounds for the problems. Lower bounds were obtained by splitting the surrogate solution into m knapsacks and solving a series of subset-sum problems using dynamic programming. The dynamic programming approach is also used to tighten the capacity constraints and obtain better upper bounds. The MULKNAP algorithm outperformed the MTM algorithm. MULKNAP was the first developed algorithm to solve instances of large size (up to n = ) in less than a second. Zeisler (2000) developed a greedy heuristic to solve an intratheater airlift problem. For an effective execution of an air operational mission, it is important to coordinate air traffic both to the location and within the location. Given a mixture of aircraft sent to the theater, the intratheater air- 32

49 lift problem considers the amount of cargo to be moved to a specific location within some specified period of time. Zeisler formulated the problem as a MKP with the objective of maximizing the throughput in a theater given a vehicle mixture and an assignment scheme with an additional constraint to even the distribution of jobs. His model finds a solution by solving the route assignment for each vehicle at each location. The model allows for a heterogeneous, user defined vehicle mix in a theater with five bed down locations and seven forward operating locations. The model preprocesses the routes by eliminating unattractive routes from the problem and then applies a greedy heuristic to select routes and assign them to the aircrafts at the bed down locations. 3.3 Multiple Choice Knapsack Problems (MCKP) MCKP Formulation MCKP is defined as a binary knapsack problem with the addition of disjoint multiple-choice constraints (Sinha and Zoltners 1979). Given m mutually disjoint classes (N 1,..., N m ) of items, pack an item from each class into a single knapsack of capacity c. Each item j N i has a profit p ij and a weight w ij. The objective of the problem is to maximize the profit of a feasible solution. The MCKP problem is formulated as: Maximize, subject to, Z = m p ij x ij (3.5) i=1 j Ni m w ij x ij c, (3.6) i=1 j Ni x ij = 1, i {1,..., m} (3.7) j Ni x ij {0, 1}, i {1,..., m}, j N i (3.8) Equation (3.5) calculates the profit of an assignment of items, a value to be maximized. Equation (3.6) ensures the single knapsack capacity is not exceeded while Equation (3.7) ensures select- 33

50 ing a single item from each of the m disjoint subsets of items. Equation (3.8) is the binary selection requirement on decision variable x ij such that x ij = 1 if selected and x ij = 0 if not selected Heuristic Solution Approaches for MCKP Sinha and Zoltners (1979) present a B & B algorithm for MCKP. The candidate problem at every branch is a MCKP. Linear Programming (LP) relaxations of the subproblems are solved for fathoming and for providing directions for branching. The strength of the algorithm lies in obtaining a quick solution to the LP relaxation of the MCKP and its efficient subsequent re-optimization as a result of branching. The algorithm was tested on problems generated randomly with coefficients from a uniform distribution. Computational results indicated that the solution times increased faster with the number of classes than with the number of variables within each class. Armstrong et al. (1983) developed a B & B algorithm to solve the MCKP. At each stage their linear relaxation was solved by one of two algorithms depending on the initialization rules: leastcost and most-cost. They developed and investigated various versions of the algorithm examining differences in data list structures, sorting techniques, and fathoming criteria. The algorithm developed eliminated considerable storage space from an earlier implementation by Sinha and Zoltners (1979) with no effect on the execution time. Armstrong et al. (1983) determined that using a heap sort algorithm reduced the computation time while solving larger problems. Pisinger (1995) presented a partitioning algorithm to derive an optimal linear programming solution, and showed how this can be incorporated into a dynamic programming algorithm so that a minimal number of classes are enumerated, sorted and reduced. This partitioning algorithm is used for solving the Linear Multiple Choice Knapsack Problem (LMCKP) and an initial feasible solution is obtained for MCKP. Starting from this initial solution, dynamic programming is used to solve the MCKP by adding new classes to the core problem as needed. This technique uses a minimal number of classes to solve MCKP to optimality. Pisinger considered problem sizes, the test instances, and the data ranges for testing his algorithm. His algorithm outperforms any known algorithm for the 34

51 MCKP. Pisinger (2001) has also modeled the budgeting problem with bounded multiple-choice constraints (BBMC) as a generalization of the MCKP. Pisinger discusses the transformation of BBMC to an equivalent MCKP using dynamic programming which leads to an increase in the number of variables, and hence proposes Dantzig-Wolfe decomposition. MCKP is solved with some extreme points in each of the transformed classes. A slope is found by determining extreme items for every class using Equations (3.10) and (3.11). M i (α) = arg max j N i p ij αw ij, i = 1,..., m (3.9) a i = arg min w ij, i = 1,..., m (3.10) j M i (α ) b i = arg max w ij, i = 1,..., m (3.11) j M i (α ) For every class N i, M i (α ) is determined where M i (α) is the set of extreme items from the class i in the negative direction of the slope. α is the optimal slope. M i (α) is found using Equation (3.9). For every class i, M i (α) may contain more than one item. Let a i and b i be the item with smallest and largest weight in the set M i given by Equations (3.10) and (3.11), respectively. Select m an item a i from every class N i and compute the residual knapsack capacity as c w iai. Exchange item a i with b i by selecting a class arbitrarily and repeat this until the new residual capacity m m c + w iai w ibi < 0 for some class i. Set x bi = c/(w bi w ai ) and x ai = 1 x bi. This is a i=1 i=1 LP-optimal solution with a i and b i being extreme items. i=1 Pisinger s algorithm is based on dynamic programming, where the search focuses on a small number of classes with a greater probability of finding an optimal solution. Kozanidis et al. (2005) introduced a variation of the LMCKP incorporating equity constraints (LMCKPE). They present a mathematical formulation of the LMCKPE and develop an optimal two phase greedy algorithm. Phase I of the algorithm enhances an existing method for LCMKP to obtain an optimal solution. This solution acts as an initial solution for Phase II which first violates 35

52 the equity constraints and then tries to reach feasibility while maintaining optimality. The algorithm terminates upon satisfying the equity constraints. Their greedy algorithm was tested on randomly generated test problems. Computational results showed that the algorithm took less time than the commercial linear programming package to solve a problem. The time savings increased as the problem size increased. 3.4 Multiple-choice Multi-dimensional Knapsack Problems (MMKP) MMKP Formulation The MMKP combines aspects of the Multidimensional Knapsack Problem (MDKP) and the Multiple- Choice Knapsack Problem (MCKP). In the MDKP, the resources are multidimensional, i.e., the knapsack consists of multiple resource constraints simultaneously satisfied. As in the MCKP, one item is selected from each class. Suppose there are m classes of items, where each class has n i items and there are l resource constraints. Each item j of class i has the non-negative profit value p ij, and requires resources given by the weight vector w ij = (w ij1, w ij2,..., w ijl ). The amount of available resources are given by a vector c = (c 1, c 2,..., c l ). The variable x ij is either equal to 0 or 1, implying item j is not picked or is picked for the ith class, respectively. The MMKP selects exactly one item from every class to maximize the total profit, subject to resource constraints. The MMKP problem is formulated as: Maximize, Z = m n i p ij x ij (3.12) i=1 j=1 36

53 subject to, m n i w ijk x ij c k, k {1,..., l} (3.13) i=1 j=1 n i j=1 x ij = 1, i {1,..., m} (3.14) x ij {0, 1}, i {1,..., m}, j {1,..., n i } (3.15) Equation (3.12) provides the profit of selecting an item from every class, a value to be maximized. Equation (3.13) ensures the resource capacity of knapsack k is not exceeded while Equation (3.14) ensures selecting a single item from each of the i classes. Equation (3.15) is the binary selection requirement on decision variable x ij such that x ij = 1 if selected and x ij = 0 if not selected. Since the MMKP is a NP-hard problem, the computation time to reach the optimal solution increases exponentially with the size of the problem (Khan et al. 2002). Such problems can be solved either to optimality or near-optimality. Near-optimal solutions are often sufficient but take less computation time than the time taken to reach the exact optimal solution. For real-time decision making applications, such as in quality adaptation and admission control of interactive multimedia systems (Chen et al. 1999), or service level agreement management in telecommunication networks (Watson, 2001 (as referenced in Hifi et al. (2004)), algorithms for finding exact solutions for the MMKP are not suitable due to the complexity of the problem and the need for rapid system response Heuristic Solution Approaches for MMKP Moser et al. (1997) developed a heuristic algorithm with the Lagrange multiplier method to solve the MMKP. The algorithm starts with the most valuable item of each class and the Lagrange multipliers are initialized to zero. The initial choice of selected items is adapted to consider the constraints and repeatedly improve with respect to the most violated constraint. The item exchanged is found by computing the increase of the Lagrange multiplier for all non-selected items in every class relative to the selected item of the class. Eventually, the item causing the least increase in the multiplier 37

54 is chosen for exchange and the multipliers are reevaluated with this least incremental value. The process is repeated until an item from every class is selected or no more replacements are possible. Once the process is complete, there may be space in the knapsack to further improve the solution by replacing some of the selected items with more valuable items. In this last phase, every item is checked to determine if it is more valuable than the selected item and does it satisfy the resource constraints. For all the exchangeable items, the item that causes the largest increase in the knapsack value is exchanged with the selected exchangable item. This process is repeated until no more exchanges are possible. On testing and comparing their heuristic against an exact algorithm using the dynamic programming approach, the Moser et al. (1997) approach yielded results in less time than the exact algorithm. Akbar et al. (2001) present two heuristic algorithms for solving the MMKP. Their initial heuristic, (M-HEU), uses both improving and non-improving item selections. The authors then extended M-HEU to an incremental method, I-HEU, for a real-time system when the number of classes in the MMKP is very large. The proposed I-HEU solves the MMKP incrementally, starting with a MMKP solution based on a smaller number of classes. The test results show that the I-HEU offers similar results as M-HEU but in less time. The scalability property of I-HEU makes it a strong candidate for real-time applications like the adaptive multimedia systems. Khan et al. (2002) solve the Adaptive Multimedia Problem (AMP) with multiple concurrent sessions, where the qualities of the individual sessions are dynamically adapted to the available resources and user preferences. A multimedia session consists of three media: audio, video, and still image. Different combinations of these three media leads to various session quality profiles. A session s quality profile is equivalent to a stack of items. The m session s quality profiles map to m classes of items. Thus, item j of class i denotes operating quality of session i. The corresponding session utility is the profit value denoted as p ij and the amount of resource k consumed by session i is denoted as w ijk. The resource constraints of the AMP are equivalent to the resource constraints of the MMKP. Finding an operating quality for each session in the AMP is equivalent to selecting one item from every class. The objective of the AMP is to maximize the system utility subject to 38

55 the system resource constraints. The Khan et al. (2002) heuristic approach (HEU) uses Toyoda s (1975) concept of an aggregate resource for selecting items for the knapsack. They improve the solution using an item exchange approach. HEU was tested and compared to branch-and-bound search using linear programming (BBLP) and Moser s heuristic (MOSER) which is based on Lagrange relaxation. The results obtained are within 6% of the optimal value obtained by the BBLP approach. HEU out-performed Moser s heuristic both in the solution-value quality and computation time. Hifi et al. (2004) propose several heuristic algorithms for the MMKP. The first two approaches are a constructive and complementary solution approach. The constructive procedure (CP) is a greedy procedure involving a DROP and an ADD phase to generate new feasible solutions from current feasible solutions. In the complementary CP (CCP) approach, they use an iterative improvement of the initial feasible solution. The CCP uses a swapping strategy; pick items and replace the item with another item selected from the same class. The algorithms were tested on Khan s (2002) benchmark problems with results indicating the CCP performed better than the CP at slightly longer computational time. The Hifi et al. (2004) third approach is the Guided Local Search (GLS) method. The GLS uses memory to guide the search to promising regions of the solution space by increasing the cost function with a penalty term applied to bad features of solutions previously visited. The authors introduce a new principle based on (a) starting the search with a lower bound obtained by a fast greedy heuristic, (b) improving the quality of the initial solution using the complementary procedure, and (c) searching for the best feasible solution over a set of neighborhoods. The GLS algorithm is a two stage procedure; the first stage penalizes the current solution while the second stage normalizes and improves the quality of the solution obtained by the first stage. Lau and Lim (2004) propose and compare two metaheuristic approaches to obtain near-optimal solutions for the logistic scheduling problem, called Available-to-Promise, modeled as a Multi- Period Multidimensional Knapsack Problem (MPMKP). The meta-heuristic approaches used to solve the MPMKP are tabu search (TS) and ant colony optimization (ACO) algorithms. The authors 39

56 use a greedy heuristic approach to find a feasible solution for each metaheuristic procedure. Their computational results on benchmark problems show that both approaches reached near-optimal solutions quickly and were within 6% deviation of the upper bound. Computationally, the ACO ran faster than the TS. Lin and Wu (2004) examine the properties of the Multiple-Choice Multi-Period Knapsack Problem (MCMPKP) based on Armstrong et al. (1982) simple dominance and convex dominance for the MCKP. The simple dominance property states that, if two items j and k from class N i satisfy w ij w ik and p ij p ik, i = 1,..., m, then item j dominates item k. The complex dominance property applies, if three items j, k, and l from class N i with w ij < w ik < w il and p ij < p ik < p il satisfy the condition (p il p ik )/(w il w ik ) (p ik p ij )/(w ik w ij ), then item k is LP-dominated by items j and l. Lin and Wu (2004) develop a heuristic approach incorporating primal and dual gradient methods to obtain a strong lower bound. Further, two branch-and-bound procedures are developed for locating the optimal solution for the MCMPKP using the lower bound as an initial solution. The first branch-and-bound procedure solves the MCMPKP using a special algorithm incorporating a concept from the nested knapsack problem while the second procedure uses the generalized upper bounding technique through the concept of multiple-choice programming. Armstrong et al. (1982) define the multiple-choice nested knapsack problem (NK) as a generalized form of the multiplechoice knapsack problem in which several resource constraints are nested across the multiple choice classes. The NK with l resources is formulated similar to MCKP represented by equations (3.5) through (3.8), except equation (3.6) is modified for several resource constraints as in equation (3.16). m w ij x ij c k, k {1,..., l} (3.16) i=1 j Ni The test results of the heuristic approach developed by Lin and Wu (2004) indicated that the lower bound for tightly structured MCMPKP is better than that for the loosely structured MCMPKP. The computational results obtained by incorporating the lower bound in the branch-and-bound pro- 40

57 cedures showed that the special algorithm performed better over the multiple-choice programming technique for the smaller sized and loosely structured MCMPKP. Parra-Hernandez and Dimopoulos (2005) developed the Hmmkp algorithm for solving the MMKP. In Hmmkp, the MMKP is first reduced to MDKP. A linear programming relaxation of the resulting MDKP is solved. Pseudo-utility and the resource value coefficients are computed and used to obtain a feasible solution for the original MMKP. Hmmkp was tested and compared against the heuristic techniques developed by Akbar et al. (2001) and found to provide better solutions compared to the other techniques. The computation time for Hmmkp was higher than the Akbar et al. (2001) heuristics as the LP relaxation employed approximately 95% of the total time. Akbar et al. (2006) transformed the MMKP. They reduce the multidimensional resource constraints to a single constraint by using a penalty vector to replace the constraints with a single representative constraint. They present a heuristic (C-HEU) by constructing convex hulls to reduce the search space to find a near-optimal solution to the MMKP. They compare the results of solving the MMKP using C-HEU to MOSER (Moser et al. 1997), M-HEU (Akbar et al. 2001), branchand-bound search using linear programming, and a greedy approach based on a linear search of all items of each group picking the highest-valued feasible item from each group (G-HEU). They tested and compared these heuristics on random and correlated problem sets. The results indicated that C-HEU yielded better results for uncorrelated data instances and had a quicker response time than MOSER and M-HEU. The solution obtained by C-HEU was between 88% and 98% of optimality. Li et al. (2004) model the strike force asset allocation problem as a problem containing multiple knapsack constraints and multiple choice constraints. This problem groups the strike force assets into packages and assigns these packages to targets and defensive assets while maximizing the total strike force potential. Li et al. (2004) develop two models considering the strike force assets to be assigned to targets and to defensive assets, respectively. The first model considers the strike force asset allocation problem without suppression using asset classes and targets. Each target is assigned at most one attack package to maximize the total damage, subject to capacity constraints 41

58 of all asset classes. The second model considers the strike force asset allocation problem with suppression by dividing the targets into targets and threats. Integer programming formulations were developed using CPLEX MIP Parallel Solver and tested using randomly generated test problems by varying the number of asset classes, targets, and threats. The results indicated that the model without suppression solved most of the problems to optimality. Li and Curry (2005) solve the multidimensional knapsack problem with generalized upper bounds (GUBMDKP) using a critical event tabu search. This problem is similar to the MMKP except the equality is replaced by an inequality in the multiple-choice constraints. This critical event tabu search has a flexible memory structure which is updated at critical events. The critical events are generated by the solutions at the constructive phase before the solution becomes infeasible and at the beginning of the destructive phase when the solution becomes feasible. The choice rules based on Lagrangian relaxation and surrogate constraint information decide the variables to be added, dropped, or swapped. The intensification and diversification is achieved by a strategic oscillation scheme visiting each side of the feasibility boundary. The trial solutions for the critical events are obtained by a local search improvement method. The three test problem instances from Li et al. (2004) were used to generate new test problem instances by incrementing the number of available variables. The computational results of the critical event tabu search on these test problems showed that the surrogate constraint choice rules achieved better solutions than the Lagrangian relaxation choice rules. The intensification in the trial solution approach reduced the average gap of the problems. Li (2005) extends the heuristic of Li and Curry (2005) by introducing a tight-oscillation process to intensify the search when the critical event tabu search approach finds trial solutions near the feasibility boundary. The trial solution approach in the critical event tabu search considered only the change in the objective function value while the tight-oscillation considers the change in the objective function value and the feasibility change in the choice rules. When comparing the two heuristics, the results showed that there is an improvement in the solution quality for the tightoscillation process. 42

59 Dawande et al. (2000) studied the multiple knapsack problem with assignment restrictions (MKPAR) and discussed approximate greedy algorithms to solve the problem. This problem considered a set of items with positive weights and a set of knapsacks with capacities, and a set of knapsacks that can hold each item while assigning each item to at most one knapsack and satisfying the knapsack capacities. The objective of the problem was to maximize the total assigned weight and minimize the utilized capacity. The first greedy approach successively solves m single knapsack problems while the second algorithm is based on rounding the LP relaxation solution. 3.5 Applications and Formulations of the MMKP-type problems Basnet and Wilson (2005) discuss an extension of the classical knapsack problem where the objective is to minimize the number of warehouses needed to store items in a supply chain considering space requirement and item storage compatibilities. The problem considered has i items to be stored in j warehouses. Each item i has a space requirement, S i, such that S i A where A denotes the space available in each warehouse. Every warehouse j has a decision variable z j set to 1 when the warehouse has one or more items or 0 otherwise. x ij is a binary variable for item i stored in warehouse j. c ik is a binary parameter for the compatibility of items i and k. The problem is formulated as: Minimize, n Z = z j (3.17) j=1 43

60 subject to, c ik = n S i x ij Az j, j = 1,..., n (3.18) i=1 x ij + x kj c ik + 1, i = 1,..., n, k = i + 1,..., n, j = 1,..., n (3.19) z j+1 z j, j = 1,..., n 1 (3.20) n x ij = 1, i = 1,..., n (3.21) j=1 { 1, if x ij and x kj compatible 0, if x ij and x kj incompatible (3.22) Equation (3.17) minimizes the number of warehouses. Equation (3.18) ensures warehouses are not overfilled. Equation (3.19) ensures incompatible items are not stored together. Equation (3.20) ensures that the lower indexed warehouses are filled before higher indexed ones. Equation (3.21) ensures that each item must be stored in one particular warehouse. The authors examined heuristics and tested them on randomly generated problem sets. The test problems generated considered the number of items, the space required for each item, and the potential incompatibility between items. Lardeux et al. (2003) formulated a sub-problem of 2-layered network design (2-LND) in the form of MMKP which they refer to as MCMK. The aim is to install capacities on links of the two layers of a network for a minimal global cost. The first layer needs enough capacity to route all traffic demand and these capacities are considered as traffic demand for a second layer. The authors modeled the problem as a 0-1 LP which is close to the generalized MCMK problem with additional constraints. Lardeux et al. (2003) formulated the problem by considering a traffic demand vector d, a capacity vector a on the first layer edges and a capacity vector b on the second layer edges. For each link e, x e and y e are the capacity of the link for the first and second layers, respectively. For each edge e = (ij) E, V e = {ve, 0 ve, 1..., ve p(e) } is the set of discrete values available for the capacity x e on link e, and γ 0 e, γ 1 e,..., γ p(e) e are the cost values on link e of the first layer. Similarly W e = 44

61 {we, 0 we, 1..., we p(e) } is the set of values in an increasing order for y e Ê with costs (δ0 e, δe, 1..., δe p(e) ). x t e and y t e are binary variables corresponding to t th value of available capacity on link e of layers 1 and 2, respectively. L 1 and L 2 are the set of indices of valid metric inequalities defining all the feasible flows. Lardeux et al. (2003) formulate the 2-LND as: Minimize, subject to, p(e) Z = γex t t e + e E t=0 e Ê q(e) δey t e t (3.23) t=0 q(e) µ l ij ij Ê t=0 p(e) λ l ij ij E t=0 wey t e t vex t t e λ l ijd ij, l L 1 (3.24) i<j p(e) µ l ij ij E t=0 vex t t e 0, l L 2 (3.25) p(e) x t e = 1, e E (3.26) t=0 q(e) ye t = 1, e Ê (3.27) t=0 x t e {0, 1}, e E, t {1,..., p (e)} (3.28) ye t {0, 1}, e Ê, t {1,..., q (e)} (3.29) Equation (3.23) calculates the total cost of installing capacities on the links, a value to be minimized. Equations (3.24) and (3.25) ensure that the resource capacity of the two layers is not exceeded. Equations (3.26) and (3.27) ensure the selection of only one variable from every class and Equations (3.28) and (3.29) are the binary selection requirements on the decision variables. In this formulation, some of the components of the constraint left-hand side matrix take negative values which make it difficult to solve this problem with any standard MCMK algorithm. To overcome this problem, the authors suggest generating new capacity constraints. Lardeux et al. (2003) developed a greedy algorithm combined with a global search process (called it ApproachMCMK), to design several quite small but difficult 2-layered networks. The performance of the ApproachMCMK was compared with two other methods they developed; OptMIP 45

62 (optimal design method obtained by solving 0-1 LP with MIP exclusively) and OptMCMK (optimal method substituting MIP for the MCMK heuristic in the last iterations). OptMIP and OptMCMK yield optimal solutions. The number of instances, the number of nodes, the number of links in the first layer, the number of links in the second layer, the number of binary variables and the number of non-negative demands were the problem parameters they employed in the problems they solved. Feng (2001) formulated the resource allocation in Ka-band satellite as a binary integer program related to the multiple choice multiple knapsack problem (MCMKP) with the objective to maximize the aggregate priority of packets arriving at all downlink spots. His formulation is: Maximize, subject to, Z = m n l p ij x ijk (3.30) i=1 j=1 k=1 m n x ijk w ij c, k = 1, 2,..., l (3.31) i=1 j=1 m i=1 j=1 k=1 j=1 n x ijk N, k = 1, 2,..., l (3.32) l n x ijk = 1, i = 1, 2,..., m (3.33) x ijk {0, 1}, i = 1, 2,..., m, j = 1, 2,..., n, (3.34) k = 1, 2,..., l where l, m, n, and N are the numbers of bursts in a round, downlink spots, transmission power levels, and antennas, respectively. w ij is the transmission power of level j for downlink spot i, p ij denotes the corresponding priority, and c is the total available power for each burst. The problem is to choose exactly one item from each class to pack in l knapsacks, such that the profit sum is maximized without exceeding any knapsack s capacity. Equation (3.30) calculates the total profit, a value to be maximized. Equation (3.31) ensures that the system power is enough to serve the selected spots with their respective power in each burst. The limited number of antenna is represented in Equation (3.32). Equation (3.33) serves every downlink spot uniquely in a round while Equa- 46

63 tion (3.34) is the binary selection requirement on the decision variable x ijk. Since the MCMKP could require long computation time to obtain an optimal solution, Feng decomposed the original problem into a sequence of l MCKPs. Chocolaad (1998) modeled the airlift loading problem of selecting a set of cargo for packing a given fleet of aircraft as a geometric knapsack problem. The formulation accounts for the shape of each item and the shape and dimensions of the containers along with the weight and volume constraints of the aircraft. Chocolaad s model considered a single knapsack of C-17 aircrafts. Romaine (1999) extends Chocolaad s research in two areas: a multiple knapsack rather than single knapsack and heterogeneous aircraft instead of homogeneous. Romaine modeled the aircraft load scheduling problem as a multidimensional multiple knapsack problem (MMDKP) with packing constraints. He models each aircraft as a knapsack with multiple constraints (weight and volume), and packing a fleet of planes as a MMDKP (space conflicts, weight and balance, hazardous cargo, floor loading). Given n items to be packed in m aircraft. Every item has a profit p j, weight w j, and volume v j associated with it. Every aircraft has a maximum weight c i that it can hold, and volume v i. The problem is to select j items to be packed into i aircraft to maximize the total profit of the selected items. This part of the problem is formulated as: Maximize, subject to, Z = m n p j x ij (3.35) i=1 j=1 n w j x ij c i, i = {1,..., m} (3.36) j=1 n v j x ij v i, i = {1,..., m} (3.37) j=1 m x ij 1, j = {1,..., n} (3.38) i=1 x ij {0, 1}, i = {1,..., m}, j = {1,..., n} (3.39) Equation (3.35) provides the total profit of packing item j in aircraft i. Equations (3.36) 47

64 and (3.37) ensure the weight and volume constraints of each aircraft are not violated while Equation (3.38) ensures each item is assigned to at most one aircraft. Finally, Equation (3.39) is the binary selection requirement on decision variable x ij such that x ij = 1 if selected and x ij = 0 if not selected. The packing constraints considered in Chocolaad s airlift loading problem are: Maximize, subject to, n Z = p j x j (3.40) j=1 s(x α ) s(x β ) φ, α β, (3.41) where α, β j = {1,..., n} s(x j ) S total, j {1,..., n} (3.42) x j {0, 1}, j = {1,..., n} (3.43) S total is the space bounding the container volume. s(x α ) and s(x β ) are the space of the α and β items, respectively. p j is the profit associated with item j. Equation (3.40) calculates the total profit by selecting item j. Equation (3.41) reflects the requirement that no two items overlap or occupy the same space. Equation (3.42) enforces the requirement that all the items be placed within the allotted area while Equation (3.43) is the decision variable for whether item j is selected or not. Due to the computational time associated with using conventional algorithms, Chocolaad (1998) and Romaine (1999) use tabu search to solve the MMDKP. Raidl (1999) presented a genetic algorithm (GA) approach for solving the multiple container packing problem (MCPP). The problem considers packing n items into m containers of equal capacity c. Each item has a profit p j and size w j associated with it. The problem is to select m disjoint subsets of items, such that each subset fits into a container and the total value of the selected items is maximized. The MCPP is formulated as a variant of the multiple knapsack problem as: 48

65 Maximize, subject to, Z = m n p j x ij (3.44) i=1 j=1 n w j x ij c, i = 1,..., m (3.45) j=1 m x ij 1, j = 1,..., n (3.46) i=1 w j c, j = 1,..., n (3.47) n w j c, j = 1,..., n (3.48) j=1 x ij {0, 1}, i = 1,..., m, j = 1,..., n (3.49) w j > 0, p j > 0, c > 0, j = 1,..., n Equation (3.44) calculates the total value of the selected items, a value to be maximized. Equation (3.45) ensures that the capacity of the containers is not exceeded while Equation (3.46) takes into account that every item is packed into a single container only. Equations (3.47) and (3.48) avoid trivial cases. Equation (3.47) ensures that each item j fits into a container or otherwise it may be removed from the problem and Equation (3.48) avoids all items fitting into one of the containers. Equation (3.49) represents the binary decision variable x ij such that x ij = 1 if selected and x ij = 0 if not selected. The MCPP is a MKP with C i = c, for all i = 1,..., m. 49

66 4. Legacy Heuristics and Test Problems Analysis 4.1 Introduction This chapter discusses in detail the legacy heuristic procedures developed for various KP variants and performs an analysis of the test problems generated for testing solution procedures for the KP variants. Section 4.2 provides a detailed stepwise outline of heuristic approaches developed for solving the Multiple Knapsack Problems (MKP), Multiple Choice Knapsack Problems (MCKP), and Multiple-choice Multidimensional Knapsack Problems (MMKP). Specifically studied are the heuristic methods proposed by Hung and Fisk (HUNGFISK, 1978), Martello and Toth (MTM, 1985), and Pisinger (PISINGERMKP, 1999b) for the MKP; Zemel (ZEMEL, 1984) and Dyer et al. (DYERKAYALWALKER, 1984) for the MCKP; Moser et al. (MOSER, 1997), Khan et al. (HEU, 2002), Hifi et al. (CPCCP and DerAlgo, 2004) and Akbar et al. (M-HEU and I-HEU, 2001) for the MMKP. Section 4.3 examines characteristics of test problems generated for KP variants. Finally, Section 4.4 examines the structure and the characteristics of the various test problems. This chapter does not examine in detail the heuristics and test problems for the Multi-Dimensional Knapsack Problems (MDKP). Details of the MDKP heuristics and test problem generation approaches are detailed in Cho (2005). 50

67 4.2 Legacy Heuristics Legacy Heuristics for Multiple Knapsack Problems (MKP) Hung and Fisk (1978) Heuristic (HUNGFISK) Hung and Fisk (1978) developed a branch and bound algorithm to solve the MKP by assigning an item to a knapsack or by excluding that item from all knapsacks. Let F be the index set of the items not assigned to any knapsack and S be the index set of the items assigned to the knapsacks. F = means a feasible solution is found and the corresponding objective function value, z, can be compared to the incumbent solution z. S F = N and S F =, where N is the index set of all items. An overview of the HUNGFISK approach follows: Step 1: Initialize z =, S =, F = N, tree-level index k = 1. Step 2: (Bounding) Solve the Lagrangian or Surrogate relaxation of the MKP with respect to F. The Lagrangian relaxation of the MKP is given by equations (4.1) through (4.3). Equations (4.4) through (4.7) represent the surrogate relaxation of the MKP. Maximize, subject to, Z = m n n m p j x ij λ j ( x ij 1) (4.1) i=1 j=1 j=1 i=1 n w j x ij c i, i = 1,..., m (4.2) j=1 x ij {0, 1}, i = 1,..., m, j = 1,..., n (4.3) Maximize, Z = m n p j x ij (4.4) i=1 j=1 51

68 subject to, m n m π i w j x ij π i c i (4.5) i=1 j=1 i=1 m x ij 1, j = 1,..., n (4.6) i=1 x ij {0, 1}, i = 1,..., m, j = 1,..., n (4.7) The Lagrange (λ j ) and surrogate (π i ) multipliers for constraints (3.2) and (3.3) of the MKP are defined by Equations (4.8) and (4.9), respectively. λ j = { p j w j (p t /w t ), if j < t 0, if j t (4.8) and π i = p t /w t, i = 1,..., m (4.9) Compute the objective value z k using the objective function of the original MKP. If z k z, go to step 5. If the solution is feasible to the original MKP, go to step 4, else go to step 3. Step 3: (Branching) Select item j and assign it to one of the knapsacks. The branching item selected depends on the type of the relaxation employed in Step 2. For the Lagrangian relaxation, select the object in F which appears in most single-knapsack solutions used in calculating Equation (4.1). In case of ties the lowest-indexed item is selected. For the surrogate relaxation, the item with the lowest-index in F is selected. Record the value of the items assigned to the knapsacks. If all items are assigned, go to step 4, else update S, F, and k and go to step 3. Step 4: Update z, its corresponding solution, and go to step 5. Step 5: (Backtracking) Find smallest tree-level k such that z k0. Denote the level preceding k 0 as k 1 = k 0 1 and corresponding object index in S as j k 1. If k 1 0, terminate the procedure, 52

69 else set k = k 1 and free all item indices in S following j k 1. Assign item j k 1 to a knapsack that is different from the knapsacks to which it had previously been assigned. Go to step 2. If j k 1 have been excluded previously, set z k 1 = and go to step 5. Martello and Toth (1985) Heuristic (MTM) Martello and Toth (1985) developed an algorithm for the MKP using an enumerative scheme where each node of the decision tree generates two branches, one that assigns an item to a knapsack and one that excludes an item from a knapsack. They denote S i (i = 1,..., m) as the stack of items currently assigned to knapsack i and S = {S 1, S 2,..., S m }. Each iteration, the MTM algorithm inserts an item j selected for knapsack i. At the end of each iteration, knapsacks 1,..., i 1 are completely loaded, knapsack i is partially loaded, while knapsacks i + 1,..., m are empty. Upper bounds are computed using surrogate relaxation and lower bounds are computed via a heuristic approach. This heuristic procedure finds an optimal solution for the first knapsack, then excluding items included in the first knapsack, finds an optimal solution for the second knapsack. This procedure continues until a constrained optimal solution for all the knapsacks is found. x ij (i = 1,..., m; j = 1,..., n) is the current solution. An overview of the MTM algorithm follows: Step 1: Initialize z = 0, S i = for i = 1,..., m, i = 1. Compute upper bound (U). Step 2: Compute lower bound (L) and the heuristic solution x. If L z, go to step 3, set z = L and x = x. If L = U, go to step 4. Step 3: (Updating) Let j = min{u : u / S i and x iu = 1}. If no such j exists, set i = i + 1. If i < m, repeat step 3, otherwise set i = m 1 and go to step 4. Insert j in S i, set x ij = 1, compute upper bound U. Step 4: (Backtracking) Let j= last item inserted in S i such that x ij = 1. If no such j exists, set S i =, i = i 1. If i = 0 stop, otherwise repeat step 4. Set x ij = 0, remove from S i all items inserted after j. Compute upper bound (U). 53

70 If U z, repeat step 4. Otherwise go to step 2. The MTM algorithm checks for the trivial cases defined in Equations (4.10) through (4.12). Equation (4.10) ensures each item j fits into at least one knapsack, otherwise it may be removed from the problem. If Equation (4.11) is violated, the smallest knapsack may be discounted and not considered for computation since no items fit into it. Equation (4.12) avoids a trivial case where all items fit into the largest knapsack. This algorithm assumes that the items are arranged in decreasing order of profit per unit weight, and the knapsacks are arranged by increasing order of capacities. max w j max c i (4.10) j=1,...,n i=1,...,m min c i i=1,...,m n w j j=1 Pisinger (1999b) Heuristic (PISINGERMKP) min w j (4.11) j=1,...,n max c i (4.12) i=1,...,m The pseudocode for the recursive branch-and-bound algorithm developed by Pisinger (1999b) is as follows: P and W are the profit and weight sums of the items assigned to a knapsack, respectively, x is the solution obtained by solving the surrogate relaxation of the MKP which is split into m knapsacks while solving a series of subset-sum problems, y ij = 0 or 1 indicates an item j assigned to knapsack i. Step 1: For i = 1,..., m and j = 1,..., n, initialize d j = 1, x ij = 0, y ij = 0, z = 0. Order the knapsacks by decreasing order of their capacities c 1, c 2,..., c m. Go to step 2. Step 2: Tighten the knapsack capacities c i by solving m subset-sum problems on items h + 1,..., n. Solve the surrogate relaxation of the MKP with x as the solution and u as the objective value. Go to step 3. Step 3: If (P + u > z ), then sort the items given by x according to non-increasing weights. Split the solution x into m knapsacks by solving a series of subset-sum problems on items j with x j = 1. 54

71 Let y ij be the optimal filling of c i with a corresponding profit sum z i. Improve the heuristic solution for i knapsacks not completely filled. Go to step 4. m m Step 4: If (P + z i > z ), then copy y to x and set z = P + z i. Go to step 5. i=1 i=1 Step 5: If (P + u > z ), then reduce the items by upper bound tests and swap the reduced items to the first positions, increasing h. Let i be the smallest knapsack with c i > 0. Solve this problem as a KP with c = c i defined on the free variables. Let x be the solution vector. Choose an item with largest profit-weight ratio among x j = 1. Denote this item as l. Swap item l to position h + 1 and set j = h + 1. Assign item j to knapsack i (y ij = 1). Recursively repeat step 2. Go to step 6. Step 6: Exclude item j from knapsack i (y ij = 0). Set d = d j and d j = i + 1. Recursively repeat step 2. Find item j again and set d j = d Legacy Heuristics for Multiple Choice Knapsack Problems (MCKP) The concept of dominance plays an important role in solving the MCKP by deleting those items from the classes that will never be chosen in an optimal solution. Two criteria of dominance as referenced by Sinha and Zoltners (1979) are: Dominance Criteria 1 If two items j and k from class N i satisfy the conditions (4.13) and (4.14), then item j dominates item k. w ij w ik, i = 1,..., m (4.13) p ij p ik (4.14) This criteria applies when some item j is more valuable and takes less resource than another item k in class i. Dominance Criteria 2 55

72 If three items j, k, and l from class N i with w ij < w ik < w il and p ij < p ik < p il satisfy the condition (4.15), then item k is LP-dominated by items j and l. (p il p ik )/(w il w ik ) (p ik p ij )/(w ik w ij ) (4.15) Zemel (1984) Heuristic (ZEMEL) Zemel (1984) developed a partitioning algorithm which relies on the property that in order to solve a MCKP, it is sufficient to find the optimal slope defined as the incremental efficiency of the last item added in the algorithm. The concept of optimal slope, as explained by Kellerer et al. (2004), are stated in Steps 3, 4 and 5 below. A slope is found by determining extreme items for every class using Equations (4.17) and (4.18). ZEMEL algorithm finds an optimal slope α which is the optimal solution for the MCKP problem. The proposition of finding an optimal solution by an optimal slope is explained below. M i (α) = arg max j N i p ij αw ij, i = 1,..., m (4.16) a i = arg min w ij, i = 1,..., m (4.17) j M i (α ) b i = arg max w ij, i = 1,..., m (4.18) j M i (α ) For every class N i, M i (α ) is determined where M i (α) is the set of extreme items from the class i in the negative direction of the slope. α is the optimal slope. M i (α) is found using Equation (4.16). For every class i, M i (α) may contain more than one item. Let a i and b i be the item with smallest and largest weight in the set M i given by Equations (4.17) and (4.18), respectively. m Select an item a i from every class N i and compute the residual knapsack capacity as c w iai. Exchange item a i with b i by selecting a class arbitrarily and repeat this until the new residual capacity c + w iai w ibi < 0 for some class i. Set x bi = c/(w bi w ai ) and x ai = 1 x bi m m. i=1 i=1 This is a LP-optimal solution with a i and b i being extreme items. 56 i=1

73 An outline of the ZEMEL algorithm is as follows: Step 1: For all classes N i, pair the items as (ij, ik). Order each item as w ij w ik breaking ties using p ij p ik. If item j dominates item k in class N i, then delete item k from class N i and pair item j with another item from class N i. Continue until all the items in class N i have been paired. For an odd number of items, one item remains unpaired. Step 2: For all classes N i, if the class has only one item j remaining, then reduce the capacity to c = c w ij, and fathom class N i. Step 3: For all the pair of items (ij, ik), compute the slope α ijk = (pik pij)/(w ik w ij ). Let α denote the median of all the slopes α ijk. Step 4: For i = {1,..., m}, derive M i (α), a i, and b i using Equations (4.16), (4.17) and (4.18). Step 5: If Step 6: If Step 7: If m m w iai c w ibi, then α is the optimal slope α. Stop. i=1 i=1 m w iai c, then for the pairs of (ij, ik) with α ijk α, delete item k. Go to step 1. i=1 m w ibi < c, then for the pairs of (ij, ik) with α ijk α, delete item j. Go to step 1. i=1 Dyer et al. (1984) Heuristic (DYERKAYALWALKER) Dyer et al. (1984) developed a branch-and-bound algorithm to solve the MCKP. The algorithm solves the LP-relaxation and reduces the classes before solving the problem through branch-andbound. An overview of the DYERKAYALWALKER heuristic is as follows: Step 1: Remove LP-dominated items by ordering the items in each class N i by increasing weights, and testing for conditions (4.13) and (4.14). Let R i contain the remaining items ordered according to w i1 < w i2,... < w iri, where r i is the size of R i. Step 2: Solve the LP-relaxation to derive an upper bound. Develop a KP instance by setting p ij = p ij p i,j 1, w ij = w ij w i,j 1 for each class R i and j = 2,..., r i. p ij is the incremental profit, a measure of the gain if item j is selected instead 57

74 of item j 1 in class R i. w ij is the incremental weight which measures an increase in the weight if item j is selected instead of item j 1 in class R i. Compute the residual capacity m c = c w i1. The indices associated here are the indices from R i. i=1 Sort the items according to decreasing incremental efficiencies ē ij = p ij / w ij. The indices associated here are the original indices. m Fill the knapsack with capacity c. Initialize z = p i1. Every time an item is inserted, set z = z + p ij, c = c w ij, x ij = 1, and x i,j 1 = 0. i=1 Let t N s be the item exceeding the knapsack capacity. Set x st = c/ w st, x s,t 1 = 1 x st, z = z + p st x st. Return LP-solution x and z. Step 3: Reduce the classes by relaxing constraint (3.8). The upper bound on the MCKP with additional constraint x ij = 1, Uij 1 is calculated using Equation (4.19). U 1 ij = p p iti + p ij + α (c w + w iti w ij ) (4.19) If Uij 1 z, then x ij = 0. Similarly test for the upper bound for constraint x ij = 0 (Uij 0 ). For this case, fix x ij = 1, other decision variables in the class are zero. If the reduced set has only one item j left, the class is fathomed fixing x ij = 1. Step 4: Solve the rest of the problem using branch-and-bound Legacy Heuristics for Multiple-choice Multi-dimensional Knapsack Problems (MMKP) Moser et al. (1997) Heuristic (MOSER) Moser et al. (1997) solved the MMKP using Lagrange multipliers. An outline of the MOSER s heuristic is as follows: 58

75 Input elements: Profit values of the items p ij, i = 1,..., m, j = 1,..., n; item weights w ijk, i = 1,..., m, j = 1,..., n, k = 1,..., l; knapsack capacities c k, k = 1,..., l Output elements: Selected elements w iρi k, i = 1,..., m, k = 1,..., l, ρ i selected item from class i Step 1: Initialization and normalization of weights Reset Lagrange multipliers µ k = 0, k = 1,..., l. Find the index of the most valuable item in each class ρ i = arg max j = 1,..., np ij, i = 1,..., m and select this element. Normalize the weights w ijk = w ijk /c k, i = 1,..., m, j = 1,..., n, k = 1,..., l. Compute the constraint violation y k = m w iρi k, k = 1,..., l. i=1 Step 2: Repeat this step until the constraint violation is resolved by exchanging the elements such that no more elements can be exchanged and y k 1, k = 1,..., l. Determine the most violated constraint by finding index k of the largest y k > 1, k = 1,..., l. Find the item to be exchanged by computing the increase δ ij of the Lagrange multiplier µ k for all the non selected items in every class relative to the selected item of the class using equation (4.20) l δ ij = (p iρi p ij µ k (w iρi k w ijk ))/(w iρi k w ijk), i = 1,..., m, j = 1,..., n k=1 Find class i and index j of the item to which smallest δ ij belongs. (4.20) Reevaluate Lagrange multipliers and constraint violations using (4.21) and (4.22). µ k = µ k + δ i j (4.21) y k = y k w iρi k + w i j k, k = 1,..., l (4.22) 59

76 Remove the selected item with index ρ i of class i. Set the item with index j as the new selected item of class i, ρ i = j. Step 3: Improve the solution by repeating this step until no more elements can be exchanged. Compute the knapsack value increase δ ij using (4.23) for all non-selected items in every class relative to the profit value p iρi of the selected element. δ ij = { p ij p iρi, 0, otherwise if p ij p iρi > 0 and y k w iρi k + w ijk 1, k = 1,..., l (4.23) Find the best exchangeable item by finding class i and index j for the largest knapsack value increase δ ij. Remove the selected item with index ρ i of class i. Select the item with index j of class i, ρ i = j. m Step 4: Compute the result y k = w iρi k, k = 1,..., l. If y k 1, k = 1,..., l, the problem can be i=1 solved and the solution is w iρi k, i = 1,..., m, k = 1,..., l. Otherwise the problem cannot be solved. Khan et al. (2002) Heuristic (HEU) Khan et al. (2002) used Toyoda s concept of aggregate resource (Toyoda 1975) for selecting items. Their heuristic has three steps as described below. Let c be the total resource usage vector, y be the current knapsack usage vector, ρ be the current solution vector, ρ/z be the solution vector after exchange Z from ρ, a be the aggregate resource savings, t be the total-per-unit-resource savings, p be the value gain per unit of extra resource, and U be the total value of the current pick. The vectors c and y contain values for knapsack utilization. Step 1: Find a feasible solution 1.1 Pick items with lowest values from every class. Let ρ i = 1, i = 1,..., m be equal to the selected m item j in every class i. Compute y k = w iρi k, k = 1,..., l. i=1 60

77 1.2 Find α where y α /c α = max k=1,...,l y k /c k. If the initial pick is feasible i.e. y α /c α 1, go to step 2, otherwise proceed to step Consider an exchange X = (i, j) from ρ picking item j from class i instead of item ρ i. Define l a(ρ, i, j) = (w iρi k w ijk ) y k / ȳ, i = 1,..., m, j = 1,..., n. Let ρ = ρ X. k=1 Find i and j such that a(ρ, i, j ) = max ij a(ρ, i, j) and y α (ρ ) < y α (ρ). if k α and y k (ρ) c k (ρ) then y k (ρ ) c k (ρ ) and if k α and y k (ρ) > c k (ρ) then y k (ρ ) c k (ρ ). If i and j are found, set ρ = ρ (i, j ) and go to step 1.2, otherwise exit the procedure with no solution found. Step 2: Improve the initial feasible solution iteratively using feasible upgrades 2.1 Consider an upgrade X = (i, j) where item j is picked from class i instead of item ρ i. Find a feasible upgrade X = (i, j ) that maximizes a(ρ, i, j). If X is found and a(ρ, i, j ) > 0, set ρ = (ρ X ) and go to step Find a feasible upgrade X = (i, j ) that maximizes p(ρ, i, j). Define p(ρ, i, j) = (p iρi p ij )/ a(ρ, i, j). If X is found and p(ρ, i, j ) > 0, set ρ = (ρ X ) and go to step 2.1. Repeat until no more feasible upgrades are possible and then go to step 3. Step 3: Iterative improvement of the upgraded solution using upgrades followed by one or more downgrade(s) 3.1 Find an upgrade Y = (i, j ) that maximizes p (ρ, i, j) where p (ρ, i, j) = (p iρi p ij )/ t (ρ, i, j) l and t (ρ, i, j) = (w iρi k w ijk )/(c k y k ). Set ρ = ρ Y. k=1 3.2 Find a downgrade Y = (i, j ) that minimizes p (ρ, i, j) such that U(ρ Y ) > U(ρ), l where p (ρ, i, j) = (p iρi p ij )/ t (ρ, i, j) and t (ρ, i, j) = (w iρi k w ijk )/(c k y k ). k=1 If Y is found and (ρ Y ) is feasible, set ρ = (ρ Y ) and go to step 2. 61

78 If Y is found and (ρ Y ) is not feasible, set ρ = (ρ Y ) and go to step Exit procedure with the solution vector ρ. Hifi et al. (2004) Heuristic (CPCCP) Hifi et al. (2004) developed a constructive and complementary search approach for solving the MMKP. The initial feasible solution is obtained using a constructive procedure (CP). The CP is a greedy approach with DROP and ADD phases. The CP is detailed as follows. Let S be the solution vector, ρ i the selected item j from class i, x iρi indicates whether or not the item from a class is selected, c k is the accumulated resources for knapsack k, and y k is the total resource used in knapsack k. Step 1: Initialization of the solution procedure 1.1 Define the pseudo-utility ratio for every item j of class i as u ij = p ij / l k=1 y kw ijk. For every class i = 1,..., m, Find item j for max{u ij, j = 1,..., n} set S i j set ρ i = j and x iρi = 1 n set y k = EndFor i=1 w iρi k Solution vector is S = (S 1,..., S m ). Go to step Main Loop DROP Phase While (y k > c k, k = 1,..., l) set k arg max k=1,...,l {y k } set i arg max i=1,...,m {w iρi k} set ρ i = j and x i ρ i = 0 62

79 Compute y k = y k w iρi k, k = 1,..., l ADD Phase For every item j = 1,..., n, If (j j and R k + w i jk < c k, k = 1,..., l) then x i j = 1 j = j ρ i = j y k = y k + w i ρ i k, k = 1,..., l S = (ρ i ; ρ i, i i, i = 1,..., m) is a feasible solution Exit with solution vector S EndIf EndWhile EndFor j i arg min j=1,...,n {w i jk } if the obtained solution is not feasible ρ i = j i ; x i ρ i = 1 Return solution vector S with value Z(S) The Complementary Constructive Procedure (CCP) tries to iteratively improve an initial feasible solution obtained using CP. It applies a local swap strategy of picked and non-picked items while maintaining feasibility. The CCP is described below. Let S be an improved solution vector with a solution value of Z(S ). Step 1: Set S = (S 1,..., S m ) from CP and set S S. Step 2: Iterative procedure calls the local swap strategy to try and improve the initial feasible solution from CP. While not StoppingCondition() do For every class i = 1,..., m, 63

80 j i LocalSwapSearch(j i, i), where j i and j i are the old and new items of class i respectively. S i j i S (S 1,..., j i,..., S m) If Z(S) > Z(S ) then S (S 1,..., j i,..., S m) EndWhile Return solution vector S with value Z(S ) The LocalSwapSearch() procedure initializes the best item to swap by setting, value p isi and s i S i, where p isi is the profit of the old item from class i and s i is a candidate item in i to be swapped. The LocalSwapSearch() procedure is: For every item j = 1,..., n, and j S i do If (p ij > value and y k w isi k + w ijk < c k, k = 1,..., l) then set value p ij set s i j Return S i as the best item for a local swap. Hifi et al. (2004) Heuristic (DerAlgo) DerAlgo, developed by Hifi et al. (2004), is a two-phase procedure using a penalty and normalized strategy. It is based on a guided local search approach. The starting point of this algorithm is the feasible solution obtained by applying CP discussed above. This solution is considered the best feasible solution obtained before the start of the algorithm without using any penalty. The algorithm uses the LocalSwapSearch() from the CCP heuristic to improve the initial solution. A penalty phase is applied if the solution does not improve after a certain number of iterations. A penalty parameter Π [0, 1] is used to transform the profit values of the items and try to 64

81 find a good neighborhood for improving the solution. The algorithm randomly picks a class and determines the index of the item from the class in the solution vector. The profit value of this item is penalized and the solution is updated. The procedure starts with a new feasible solution for the original problem and finds a good neighborhood to try to improve the current solution. This helps the search procedure diversify the search space by releasing the current solution from the local optima and modify the search trajectory. Normalization is used to transform the penalized solution to a normal feasible solution by setting the profit values of the current solution to the original profit values. The penalty and normalization phases are summarized below. Let 0 < Π < 1 be the penalty coefficient, ρ be the current solution vector in the penalized phase, be the depth parameter for penalization, and D be the diameter parameter for exploration. The depth parameter initiates the number of items to penalize. The diameter parameter controls the search space of the better obtained solutions up to now. An outline of the DerAlgo is as follows: Step 1: Initialization of the solution procedure Set S S := CP () and V (ρ) Z(S ). Initialize the depth, diameter, and phase parameters, D, and phase respectively as = D 0, and phase N ormal P hase. Step 2: Main Step using the two-phase procedure of penalization and normalization of solution While not StoppingCondition() do S:=CCP(S) If V (ρ) Z(S) then If (phase = Normal P hase) then Set S S and V (ρ) Z(S ) Else Set S Normalize(S, ρ, Π) Set S S and V (ρ) Z(S ) EndIf Else 65

82 If (phase = Normal P hase) then S P enalize(s, V (ρ), Π, ) Else Set S Normalize(S, ρ, Π) and V (ρ) Z(S) S P enalize(s, V (ρ), Π, ) EndIf EndIf Increment(D) Set Get Depth(, D, n) EndWhile Return S with value Z(S ) The Penalize() procedure is as follows: Step 1: Initialization of Penalize() procedure Set the initial solution S S and V (ρ) Z(S). Set Counter = 0, where Counter is the variable used to control the depth parameter (0 Counter ). Step 2: Main loop for Penalize() procedure While (Counter ) do Random selection of a class i GetClass()) j i ρ i p iρi Π p iρi Z(S) Z(S) p iρi + Π p iρi Increment(Counter) EndWhile Return S with value Z(S) as the penalized current solution 66

83 follows: The Normalize() procedure normalizes the improved penalized solutions. This procedure is as Step 1: Initialization of Penalize() procedure Set the initial solution S S and V (ρ) O(S ). Step 2: Main loop for Penalize() procedure For i = 1,..., m do j i ρ i O(S ) O(S ) p ij + (1/Π) p iρi p iρi = (1/Π) p iρi EndWhile Return S with value O(S) as the penalized current solution Akbar et al. (2001) Heuristic (M-HEU and I-HEU) The heuristic developed by Akbar et al. (2001) sorts the items of each class in an increasing order of profit values. The lower-valued items are at the top of every class while the higher-valued items are at the bottom of every class. An infeasibility ratio is defined as f k = y k /c k where y k is the amount of k th resource consumption denoting the sum of resources required by the picked items and c k is the amount of available resources. The k th resource is feasible if f k 1. An outline of M-HEU heuristic follows: Step 1: Find a feasible solution 1.1 Select the lowest valued item from every class. 1.2 If the resource constraints are satisfied, go to step 2, otherwise the current solution is infeasible. 1.3 Find the resource k l = arg max f k. Select a high valued item from any group which decreases f kl, does not increase the infeasibility factor of other resources, maintains feasibility, and yields the 67

84 highest aggregate resource consumption given by Equation (4.24). With w ijk as the amount of k th resource consumption of the j th item of i th class, ρ i as the index of selected items from i th class, and y k as the amount of the k th resource consumption, define l a ij = (w iρi k w ijk ) y k / ȳ (4.24) k=1 p ij = p iρi p ij (4.25) 1.4 If an item is found in step 1.3 then go to step 1.2, otherwise no solution is found. Step 2: Upgrading the selected feasible solution 2.1 Find a higher valued item from a class, other than the selected item of that class, subject to the resource constraint with the highest positive value of a ij. If no such item is found then an item with the highest p ij / a ij is chosen. 2.2 If no such item is found in step 2.1 then go to step 3, otherwise check for other items in step 2.1. Step 3: Find one upgrade followed by at least one downgrade 3.1 If there are higher valued items than the selected item in any class, then find such a higher valued item with the highest value of p ij / a w ijk, where a ij is the ratio of increased resource requirement to available resource and is given by Equation (4.26). a ij = l (w iρi k w ijk )/(c k y k ) (4.26) k=1 3.2 Find a lower-valued item than the selected item of the classes such that the downgrade gives higher total value than the total value obtained from step 2 and has the highest value of a ij / p ij, where a ij is the ratio of decreased resource requirement to over consumed resource and is given by Equation (4.27). 68

85 a ij = l (w iρi k w ijk )/(y k c k ) (4.27) k=1 3.3 If an item is found in step 3.2 and if the item satisfies the resource constraint then check for a better solution with step If an item is found in step 3.2 and if the item does not satisfy the resource constraint then downgrade with step If no such item is found in step 3.2 then store the solution found using step 2 and terminate. As the number of classes increases, M-HEU is not efficient. The authors propose an incremental heuristic I-HEU to solve the MMKP incrementally from an existing solution of the MMKP with smaller number of classes. I-HEU consists of similar steps as M-HEU with modifications in step 1. The modifications in step 1 are as follows: 1.1 Select the lowest valued item from every new class. 1.3 This step is similar to step 1.3 of M-HEU except that any item can be found instead of a higher valued item. 4.3 Test Problem Analysis A fundamental contribution of this research is the systematic use of test problems to gain heuristic performance insight. In this section, existing test problems and test generation approaches are examined to characterize the problems and highlight shortcomings in existing problems. Test problem structure is examined in the next subsection Test Problem Analysis for Multiple Knapsack Problems (MKP) Hung and Fisk (1978) generated test problems for the MKP consisting of up to 200 items and up to 6 knapsacks. The profit values and the weights of the items were independently generated from a 69

86 discrete uniform distribution (10, 100). The knapsack capacities c i were generated from the interval c l c i c u based on Equations (4.28) and (4.29). c l = [0.4( c u = [0.6( n w j /m)] (4.28) j=1 n w j /m)] (4.29) j=1 Hung and Fisk (1978) generated the final knapsack capacity c m such that the occupancy ratio= m i=1 c i/ n j=1 w j = 0.5. The generated capacities are discarded and a new set of capacities are generated if c i < min j w j or max i c i < max j w j. The authors used 0.5 as the occupancy ratio for all the problems generated. Martello and Toth (1985) generated test problems by independently generating the profit p j and weight w j from a uniform distribution in the interval (10, 100), and the capacity c i values from a uniform distribution satisfying the conditions (4.30) and (4.31). 0 c i 0.5( c m = 0.5 n i 1 w j ) c u, i = 1,..., m 1 (4.30) j=1 n j=1 u=1 m 1 w j c u, i = 1,..., m 1 (4.31) u=1 The authors varied the number of items n (n = 50, 100, 200, 500, 1000), the number of knapsacks m (m = 2, 5, 10) and generated 10 problems for each setting of n and m. Pisinger (1999b) randomly generated four different types of test problem instances using range limits R = 100, 1000, and These four test problem instances were then classified into: Uncorrelated test problem instances: Profit p j and weight w j of item j are randomly distributed in the interval (10, R). 70

87 Weakly correlated test problem instances: Weight of item j (w j ) is randomly distributed in the interval (10, R), and the profit of item j (p j ) is randomly distributed in the interval (w j R/10, w j + R/10) such that p j 1. Strongly correlated test problem instances: Weight of item j (w j ) is randomly distributed in the interval (10, R), and profit of item j (p j ) is set to w j Subset-sum test problem instances: Weight of item j (w j ) is randomly distributed in the interval (10, R), and profit of item j (p j ) is set equal to w j. Pisinger (1999b) considered two different classes of capacities: Similar capacities having the first m 1 capacities c i randomly distributed in the interval given by Equations (4.28) and (4.29). Dissimilar capacities having the capacities c i distributed in the interval given by equation (4.32). n i 1 0 c i 0.5( w j c u ), i = 1,..., m 1 (4.32) j=1 u=1 For both the classes of capacities, the capacity of the last knapsack c m is chosen as Equation (4.31) ensuring that the sum of capacities is half the total sum of the weights of items. A new test problem instance is generated if Equations (4.10) through (4.12) are violated. Table 4.1 summarizes the design in various studies of MKP heuristics involving test problem generation. The studies focus on computational time all but ignoring solution quality and the noncomputer specific computational work measured by number of iterations Test Problems for Multiple Choice Knapsack Problems (MCKP) 71

88 Table 4.1: Factors and Measures used in Empirical Analysis of the MKP Heuristics Author Factors Measures m n D S Σ Tm Acc/Err OpS Iter Hung and Fisk (1978) x x x x x x Martello and Toth (1985) x x x x Pisinger (1999b) x x x x x m= number of knapsacks n= number of items D= Distribution of constraint coefficients S= Slackness of constraints Σ=Correlation induced between problem coefficients Tm= CPU time taken to solve the problem Acc/Err= Accuracy or error between heuristic solution value and optimal solution value OpS= number of problems solved to optimality Iter= number of iterations Sinha and Zoltners (1979) randomly generated test problems for the MCKP. They generated the profit (p ij ) and the weight (w ij ) values using a uniform distribution, ensuring that the same profit values were not repeated within a class, and the same weight values were not repeated within a class. They ordered the profits and the weights in an increasing order ensuring that the test sets had no dominated items. The knapsack capacity was generated using Equation (4.33). Dyer et al. (1984) randomly generated test problems using Sinha and Zoltner s (1979) MCKP problem generation method. The objective and the constraint coefficients were independently generated from a uniform distribution on an interval with a lower bound of 100 and a fixed width. The generator prevented the repetition of the coefficients within the multiple-choice set. The right-hand side of Equation (4.33) was calculated as: 72

89 m c = (0.5) (min(a j ) + max(a j )) (4.33) j N i j N i i=1 Pisinger (1995) generated test problem instances with data in the range R = 1000 or 10000, varying the number of classes m and number of items N i to be packed in a knapsack. He developed five test instances by calculating the capacity using Equation (4.33). Uncorrelated test problem instances: N i items in each class are generated by choosing weights w ij and profits p ij randomly from the interval (1, R). Weakly correlated test problem instances: For every class i, weight of item j (w ij ) is randomly distributed in the interval (1, R), and profit of item j (p ij ) is randomly distributed in the interval (w ij 10, w ij + 10) such that p ij 1. Strongly correlated test problem instances: For every class N i items are generated with weights and profits (w ij, p ij ). w ij are randomly distributed in interval (1, R) and p ij = w ij These items are ordered by increasing weights. The weights and the profit values of the test problem instances for the MCKP are then generated by Equations (4.34) and (4.35) ensuring no dominated items. w ij = h ij = j w h, j N i (4.34) h=1 j w h, j N i (4.35) h=1 Subset-sum test problem instances: These test problem instances have w ij randomly distributed in the interval (1, R) and have p ij = w ij. Sinha and Zoltners: Pisinger (1995) developed test instances using Sinha and Zoltner s (1979) method by selecting the profit values and the weights from a randomly distributed interval (1, R) and arranging them in an increasing order ensuring no dominated items in the test problem instances. 73

90 Kozanidis et al. (2005) generated their test problems using Sinha and Zoltners (1979) MCKP problem generation method. The profit and weight values were generated from a uniformly distributed interval (0, R), where R was varied from 100 to 500. Their test problems contained dominated and non-dominated items within a multiple-choice set. Table 4.2 summarizes the design of various studies of the MCKP heuristics in which test problems were generated. As in the MKP instances, the measures used in the empirical analyses are not particularly beneficial. Table 4.2: Factors and Measures used in Empirical Analysis of the MCKP Heuristics Author Factors Measures m n i D S Σ Tm Acc/Err OpS Iter Sinha and Zoltners (1979) x x x x Armstrong et al. (1983) x x x x Pisinger (1995) x x x x x Kozanidis et al. (2005) x x x m= number of classes n= number of items in each class D= Distribution of constraint coefficients S= Slackness of constraints Σ=Correlation induced between problem coefficients Tm= CPU time taken to solve the problem Acc/Err= Accuracy or error between heuristic solution value and optimal solution value OpS= number of problems solved to optimality Iter= number of iterations Test Problems for Multiple-choice Multi-dimensional Knapsack Problems (MMKP) 74

91 Moser et al. (1997) developed MMKP test problems by first considering a MDKP of n items with weights and profits randomly selected from the interval generated between 1 and 20. They then transformed the MDKP to MMKP by dividing the items into m classes, each class containing at least one item. Condition (4.36) was ensured and the test problem generation was restarted if the condition was not satisfied. They varied the number of classes from 2 to 9, the number of knapsacks from 1 to 5, and the number of items in every class from 1 to 100. n i w ij c i w ij, i {1,..., m} (4.36) j=1 Akbar et al. (2001) randomly generated MMKP test instances. The pseudo random numbers initialized were: kth weight of jth item belonging to ith class, w ijk =random(0, (R c 1)) Value per unit resource k, p k =random(0, (P c 1)) Value of every item, v ij = k w ijk p k +random(0, (V c 1)) Resource capacity of kth knapsack is generated as, c k = R c m 0.5, where m is the number of classes Selected item of ith class, ρ i =random(0, (C i 1)), where C i =number of items in ith class Resource capacity of kth knapsack is exactly equal to the sum of the resources of the selected items, c k = i w iρi k Profit values of the selected items, p iρi = k w iρi kp k + V c R c, P c, and V c are upper bounds on resource requirement, unit price of any resource, and extra profit value of an item after its consumed resource price, respectively. The authors randomly generated 10 data sets for each of the parameters l, m, and n by setting R c = 10, P c = 10, and V c =

92 Khan et al. (2002) generated random uncorrelated and correlated MMKP test problem instances. For uncorrelated data instances the profit values of the jth item of the ith class, v ij were generated randomly by a random(0, (m R c /2 P c /2)) (j +1)/l function. For correlated data instances, these values were generated by the v ij = w ijk P k random(0, (m 3 R c /10 P c /10)) function where R c and P c are the maximum amount of a resource consumption by an item and maximum cost of any unit resource, respectively. The interval inside the random() function denotes the range used to generate the random numbers from a uniform distribution. These instances were generated by varying the number of classes from 5-30 and The number of items in each class and number of knapsacks considered were 5 and 10. Table 4.3 summarizes the design of various studies of the MMKP heuristics in which problem instances were generated. In these studies, the authors did relate heuristic quality in terms of nearness to an optimal. 4.4 Problem Structure Analysis of Test Problems This section examines in some detail the structure of the various test problems employed in past studies of KP-variant heuristic performance Structure of MDKP Test Problems Beasley (2006) provides a set of 48 test problems for the Multi-Dimensional Knapsack Problems from literature. Hill (1998) first analyzed the structure of these 48 test problems. The data available includes the number of knapsacks, number of items, knapsack capacities, and the optimum solution. Correlation values between the objective function and the constraint coefficients and the interconstraint correlation are calculated and plotted in Figures 4.1 and 4.2. Figure 4.1 plots the range of objective function to constraint coefficient correlation values for the 48 test problems referenced across the X-axis. There are more problem instances with 76

93 Table 4.3: Factors and Measures used in Empirical Analysis of the MMKP Heuristics Author Factors Measures l m n D S Σ Tm Acc/Err OpS Iter Moser et al. (1997) x x x x x x Akbar et al. (2001) x x x x x x x x Khan et al. (2002) x x x x x x l= number of knapsacks m= number of classes n= number of items in each class D= Distribution of constraint coefficients S= Slackness of constraints Σ=Correlation induced between problem coefficients Tm= CPU time taken to solve the problem Acc/Err= Accuracy or error between heuristic solution value and optimal solution value OpS= number of problems solved to optimality Iter= number of iterations negative correlation and the average correlation values centered around zero. These problems have 5 knapsacks, items from 30 to 90, and constraint coefficient correlation ranging from -0.2 to 0.1. The coefficient correlation ranges of the problems are narrow and not variable given the entire range of correlation values. This would be insufficient to gain an insight about the heuristic performance as influenced by the correlation structure since the ranges are limited, and many are simply due to random sampling. Figure 4.2 shows the interconstraint coefficient correlation ranges for the above 48 test problem instances. As observed for the correlation between objective function and the constraint coefficients, the interconstraint coefficient correlation varies around zero. Once again the range of interconstraint correlation structures is limited and due to mostly sampling variation versus systematic control in 77

94 Figure 4.1: Range of Correlation Values Between Objective Function and Constraint Coefficients for MDKP Standard Test Problems the problem generation procedure. Figure 4.2: Range of Correlation Values Between Constraint Coefficients for MDKP Standard Test Problems Structure of MKP Test Problems Hung and Fisk s (1978) MKP test problem instances are unavailable, so a sample was generated using their problem generation method. The number of items were varied as n = 20, 30, 40, 60, 80, 100 and the number of knapsacks were varied as m = 2, 3, problem files each containing 10 prob- 78

95 lems were generated for each of the combinations of n and m. Correlation values between the objective and the constraint coefficients were calculated and are plotted in Figure 4.3. Figure 4.3 plots the range of objective function to constraint coefficient correlation values for each test problem file generated. The correlation coefficient ranges of these problems are narrow ranging from -0.6 to 0.3. These average correlation values are centered around zero and do not provide a very large range. The range observed is due to sampling error given the relatively small number of items considered. Figure 4.3: Range of Correlation Values Between Objective Function and Constraint Coefficients for Hung and Fisk MKP Test Problems A sample of Martello and Toth (1985) test problems were generated by varying the number of items as n = 50, 100, 200 items and the number of knapsacks as m = 2, 5, 10, to produce 9 problem files each containing 10 problems. The correlation coefficient values between the objective and the constraint coefficients are plotted in Figure 4.4. The correlation values are distributed from to 0.25 which is a very narrow range. The average correlation values are nearly zero for all the problem files. Pisinger (1999b) MKP test problem instances were generated for uncorrelated, weakly correlated, strongly correlated, and subset-sum problems. The test problems were generated by varying the range limit R = 100, 1000, 10000, the number of items as n = 25, 50, 75, and the number of knapsacks m was set to 5 yielding 9 problem files with 100 problems each. Pisinger (1999b) MKP 79

96 Figure 4.4: Range of Correlation Values Between Objective Function and Constraint Coefficients for Martello and Toth MKP Test Problems problems were also analyzed by calculating the correlation values from the generated test problems and via theoretical properties using a population correlation quantification technique from Reilly (2006). The MKP is a generalization of the KP such that a subset of items is selected for every knapsack so that each of these subsets satisfies that particular knapsack. The expected population correlation for 0-1 KP can be quantified as in Equation (4.37). α Corr(w j, p j ) = 2 1 α 2 + 4δ(δ + 1) 1 (4.37) where α is the range of the distribution, δ and γ are the nonnegative integers. γ = 0 for weakly correlated coefficients, δ = 0 for strongly correlated coefficients, and δ = 0 and γ = 0 for subsetsum problems. The population correlation was calculated using Equation (4.37) by varying α = 100, 1000, and δ and γ according to problem type. The correlation values calculated by Reilly (2006) method and by computation are tabulated in Table 4.4. The correlation coefficient between the objective and the constraint coefficients for weakly correlated problems is approximately 0.97 for all the range limits. The correlation is perfect for 80

97 Table 4.4: Theoretical and Practical Objective and Constraint Coefficient Correlation for Pisinger s MKP Test Problems Problem Type α δ γ Theoretical Corr(w j, p j ) Observed Corr(w j, p j ) Weakly Correlated Strongly Correlated Subset-Sum strongly correlated and subset-sum problems. These MKP test problem instances fail to have any diversity in problem structure Structure of MCKP Test Problems Sinha and Zoltners (1979) MCKP test problems were generated varying the range limit R = 1000, 10000, number of classes m = 5, 10, 20, and number of items in each class N i = 5, 10 with 20 problem instances generated for each combination. The correlation coefficient values between the objective and constraint coefficients were calculated and are plotted in Figure 4.5. The correlation values varied between 0.5 and 0.8 and are narrow with respect to the entire range of correlation values. The problems generated with this method lack correlation structure diversity. Since the MCKP is defined as a 0-1 KP with the addition of disjoined multiple-choice constraints (Pisinger 1995), the problem structure for Pisinger (1995) MCKP test problems were analyzed via theoretical properties using Reilly (2006) population correlation quantification technique 81

98 Figure 4.5: Range of Correlation Values Between Objective Function and Constraint Coefficients for Sinha and Zoltners MCKP Test Problems for 0-1 KP. The correlation values are tabulated in Table 4.5 for different range limits. Once again this problem generation technique is fairly limited. The correlation values between the objective and the constraint coefficients for weakly correlated problems is approximately 0.98 for all the range limits. The correlation value is a perfect one for strongly correlated and subset-sum problems Structure of MMKP Test Problems Moser et al. (1997) MMKP test problems were generated by varying the number of classes as m = 5, 10, 20, the number of items in each class as n = 5, 10, and the number of knapsacks as l = 5, 10 with 10 problems generated for each combination. Figure 4.6 depicts the correlation coefficient ranges for between objective and constraint coefficients. The correlation values are distributed in the range of -0.5 and 0.5 and the average correlation value is centered around zero. While seemingly a large range, these values can be attributed to sampling variation. Khan et al. (2002) standard test problems are available at Hifi (2006). The data available for each of the test problem instances include the number of resources, number of classes, number of items in each of the classes, number of constraints, resource capacities, exact solution values, Moser et al. (1997) and the Khan et al. (2002) heuristic approaches. Correlation values between the 82

99 Table 4.5: Theoretical and Practical Objective and Constraint Coefficient Correlation for Pisinger s MCKP Test Problems Problem Type α δ γ Theoretical Corr(w j, p j ) Observed Corr(w j, p j ) Weakly Correlated Strongly Correlated Subset-Sum objective and the constraint coefficients and correlation values between the constraint coefficients are calculated and summarized in Tables 4.6 and 4.7, respectively. Figures 4.7 and 4.8 plot the analysis of the test problems. There are 13 test problems provided by Khan et al. (2002), in files labeled I01 to I13. The Figure 4.7 ranges on the smaller sized problems, I01 to I04, are fairly wide but just on the positive side. The larger sized problems, I07 to I13, have smaller correlation ranges. As the problems get larger, and involve more variables, the sample correlation structures start converging to zero, thus giving the tighter ranges and typically easier problems as problem size increases. Figure 4.8 shows the range of interconstraint coefficient values for the 13 test problems. The correlation values are evenly distributed on both the positive and the negative side of the correlation axis, are centered around the zero correlation level, and have no effective range. Correlated test problem instances were generated using the Khan et al. (2002) test problem generator. 24 test problems were generated by varying the number of classes from 5 to 40, number 83

100 Figure 4.6: Range of Correlation Values Between Objective Function and Constraint Coefficients for Moser s MMKP Test Problems of items as 5, 7, and 9, and number of knapsacks set as 5. Further examination of these problems for the correlation structure between the objective function and the constraints and the interconstraint correlation are plotted in Figures 4.9 and 4.10, respectively. Near zero objective function to constraint correlation is observed; in some cases their uncorrelated approach generates a larger range of correlation values. Interconstraint correlation hovers around zero meaning the problems effectively involve independent sampling. 4.5 Summary The knapsack problems and its variants are a difficult class of combinatorial optimization problems. Various heuristic and exact general approaches have been developed to solve these problems. Greedy heuristic approaches often provide a good bound for exact approaches, can find good initial solutions, and are used to improve the solution quality. Many researchers begin solving a KP by finding an initial solution, propose some main method which uses various greedy approaches as a base heuristic, and then use an improvement phase to obtain a final solution. Although such heuristics have shown competencies based on published computational tests, heuristic performance is really test problem dependent and problem specific. The generality of empirical test results is 84

101 Table 4.6: Correlation Between Objective and Constraint Coefficients Analysis of Khan s MMKP Test Problems Problem File min max (m,n,l) I (5,5,5) I (10,5,5) I (15,10,10) I (20,10,10) I (25,10,10) I (30,10,10) I (100,10,10) I (150,10,10) I (200,10,10) I (250,10,10) I (300,10,10) I (350,10,10) I (400,10,10) (m,n,l) represents (classes, items, resources) in problems only as valid as the test problem suite is representative of actual problems. This chapter detailed legacy heuristics for KP variants and conducted analyses of test problem structure on available test sets, or on the documented problem generation method. Such analyses have not been previously accomplished. The results of the test problem analyses should raise concerns. In general, test problems are not particularly varied in terms of structure and have characteristics that are achieved randomly. This raises concerns regarding the generality of past solution results to actual problem instances. To overcome these limitations, this research next develops a test problem set varying the above mentioned problem attributes, but with more emphasis given to controlling correlation structure and 85

102 Figure 4.7: Range of Correlation Values Between Objective Function and Constraint Coefficients for Khan s MMKP Test Problems slackness settings or at least obtaining structures over a reasonable range of values. The remaining research focuses on the MMKP. The knowledge gained by conducting an empirical analysis of various heuristic methods on these test problems is used to develop new greedy heuristics and tabu searches. Legacy methods and new methods are tested and compared using existing test problem sets as well as new problem test sets. 86

103 Table 4.7: Interconstraint Correlation Coefficients Analysis of Khan s MMKP Test Problems Problem File min max (m,n,l) I (5,5,5) I (10,5,5) I (15,10,10) I (20,10,10) I (25,10,10) I (30,10,10) I (100,10,10) I (150,10,10) I (200,10,10) I (250,10,10) I (300,10,10) I (350,10,10) I (400,10,10) (m,n,l) represents (classes, items, resources) in problems 87

104 Figure 4.8: Range of Correlation Values Between Constraint Coefficients for Khan s MMKP Test Problems Figure 4.9: Range of Correlation Values Between Objective Function and Constraint Coefficients for the correlated MMKP Test Problems generated from Khan s MMKP Test Problem Generator 88

105 Figure 4.10: Range of Correlation Values Between Constraint Coefficients for the correlated MMKP Test Problems generated from Khan s MMKP Test Problem Generator 89

106 5. Empirical Analyses of Legacy MMKP Heuristics and Test Problem Generation 5.1 Introduction Heuristics are approximate solution techniques and have long been used to solve difficult combinatorial problems. Heuristic optimization algorithms seek good feasible solutions to optimization problems where the complexity of the problem or the limited time available for its solution does not allow obtaining an exact solution. Unlike exact algorithms, where time-efficiency and resourceusage are main measures of success, two important issues that arise while evaluating heuristics are: how fast solutions are obtained and how close they come to optimal (Rardin and Uzsoy 2001). According to Hill (1998), heuristic solution procedures allow a modeler to work with more complex but often more interesting problems. Heuristic algorithms are developed and tested for three reasons: to find a feasible solution for a problem that was not previously solved, to improve the performance over existing algorithms, and to compare performance to understand how heuristics perform on different classes of problems (too often research ignores the study of this aspect of heuristic design) (Hill 1998). Characteristics of the problem being solved can influence the performance of a heuristic algorithm. These performance influences are recognized but not fully understood. This research seeks performance knowledge useful for solving the MMKP explicitly using empirical analyses as a means to gain heuristic performance knowledge. As mentioned in Chapters 3 and 4, there are different formulations of the variants of the knap- 90

107 sack problems. For the standard form of the KP variants, the problem characteristics examined are number of classes, number of items of each class, and total number of items of each instance. For MDKP, there are three important indicators of problem difficulty: the number of variables in the problem, the number of constraints in the problem, and the tightness between the constraints (Cho et al. 2003b). Hill and Reilly (2000) found that coefficient correlation structure and constraint slackness settings affected solution procedure performance in a two-dimensional knapsack problem. They found that population correlation structure, particularly interconstraint correlation, is a significant factor influencing the performance of solution methods. Additionally, the interaction between constraint slackness and population correlation structure also influences heuristic performance. No comparable problem characteristic insight exists for the other KP variants. This chapter discusses problem generation and develops a test problem generation method for the MMKP. It also provides an empirical analyses of the heuristic solution procedures for the MMKP. The legacy heuristic approaches proposed by Moser et al. (MOSER, 1997), Khan et al. (HEU, 2002), and Hifi et al. (CPCCP and DerAlgo, 2004) for the MMKP are compared on Khan s 13 available test problem instances, a small set generated per an approach proposed by Khan, and on newly generated MMKP test sets. 5.2 Problem Generation and Problem Characteristics for MMKP The previous chapter examined the lack of diversity in the problem characteristics of existing MMKP test problem instances. Empirical studies compare heuristic performance using test problems. Researchers can use available test problems (such as those available via the internet (Beasley 2006)) or randomly generated test problem instances to verify their algorithm performance. Since the existing standard MMKP test problem instances are not particularly diverse in the range of problem characteristics, the experimental information gained using these sets to examine heuristic performance is necessarily restricted. This section discusses the current MMKP test problem gener- 91

108 ation approach, a new test problem generation methodology for the MMKP test problem instances using the Cho (2005) competitive MDKP test sets, and an extended set of new MMKP test sets. The new MMKP test sets generated from the problem generation schemes discussed below can be used to study the performance of the legacy heuristics and draw performance conclusions based on a wider range of test problems Standard MMKP Test Problem Generation The thirteen test problems generated by Khan et al. (2002), available at Hifi (2006), have become somewhat of the standard MMKP test problems. The problem generation scheme for these problems is summarized below. Khan et al. (2002) generated random and correlated MMKP test problem instances. For uncorrelated data instances, the profit values of the jth item of the ith class, v ij, were generated randomly by a random(m R c /2 P c /2) (j + 1)/l function. For correlated data instances, these values were generated by the v ij = w ijk P k random(m 3 R c /10 P c /10) function where R c and P c are the maximum amount of resource consumption by an item and maximum cost of any unit resource, respectively. These instances were generated varying the number of classes from 5-30 and The number of items in each class and number of knapsacks considered were 5 and 10. Khan s problem generation approach was mimicked to generate 24 test problems varying the number of classes from 5 to 40, number of items as 5, 7, and 9, and number of knapsacks set as 5. The resulting problem correlation structure for the MMKP test problems was studied in Subsection That analysis indicated that the range of the correlation values between the objective and constraint coefficients is very narrow. The interconstraint correlation values averaged zero. This correlation range is far too narrow to generalize any performance conclusions to the potentially wide range of actual problems. All the test problems also have the same constraint capacity values. It is doubtful actual problems have identical constraints. Thus, these test problems are inadequate both in quantity and in the diversity in problem structure. What is needed is a more diverse and adequate 92

109 MMKP test set. The following subsections discuss the new MMKP test problem set generation methods developed by varying the problem attributes emphasizing in particular problem correlation structure and constraint capacity settings. The first method generates problems useful for empirical analysis aimed at understanding heuristic performance. These are called analytical problems and parameter levels are systematically varied. The second method generates problems useful for computational testing of heuristics. These are called competitive problems and parameter levels are selected to randomly cover a desired range. Each method starts with a base set of MMKP problems whose primary characteristic is a wide range of correlation structures but whose right-hand side values are not set; these values are set systematically according to defined experimental settings to produce the analytical set Analytical MMKP Test Problem Generation The base MMKP test set was generated using the Cho (2005) competitive test sets. These test sets were generated by Cho (2005) with a wide range of correlation structure ( 0.9, 0.9) and varied constraint right-hand side values. This test set overcomes test problem limitations associated with the MDKP. The problem generation method for the competitive MDKP test set is provided in Appendix B. The base MMKP test sets were generated by varying the number of classes as 5, 10, 25; the number of knapsack constraints as 5, 10, 25; and fixing the number of items in every class to 10. Nine test files containing 30 problems each, giving a total of 270 test problem instances were generated. Each file corresponds to a different combination of number of classes and number of knapsack constraints. The file naming of the different combinations are: 5GP10IT5KP, 10GP10IT5KP, 25GP10IT5KP, 5GP10IT10KP, 10GP10IT10KP, 25GP10IT10KP, 5GP10IT25KP, 10GP10IT25KP, 25GP10IT25KP where GP represents number of classes, IT represents number of items in every class, and KP represents number of knapsack constraints. The 3870 analytical test problems for the MMKP were generated by setting the right-hand side 93

110 Table 5.1: Coded Combinations of Slackness Settings for 5 Knapsacks Combination S 1 S 2 S 3 S 4 S values of the knapsack constraints in the base problem sets. Two constraint tightness levels were considered, loose and tight, coded as 0 and 1 where 0 indicates the loose setting and 1 indicates the tight setting. Of the 2 5 = 32 combinations for 5 knapsacks, 6 combination settings were considered. Combination 1 represents all the knapsack constraints loose while Combination 6 represents all the knapsack constraints tight. Similarly, 11 and 26 combinations were considered for 10 and 25 knapsacks, respectively. This reduction in design settings follows the practice established in Cho (2005). Tables 5.1, 5.2, and 5.3 show the different combinations of the constraint tightness settings. For each of the three problem sets with 5 knapsacks, 180 problems were generated. For the three problem sets with 10 knapsacks, 330 problems were generated. For the three problem sets with 25 knapsacks, 780 problems were generated. Thus, a total of 3870 MMKP problem instances were generated. The tightness levels for a constraint, S k, were set to 0.6 for tight and 0.9 for loose. The righthand side values for knapsack k were set using Equation (5.1). c k = S k [max {value of constraint coefficient of k} + (m 1) range] (5.1) where m is the number of classes and range is the range of the values of the constraint coefficients. 94

111 Table 5.2: Coded Combinations of Slackness Settings for 10 Knapsacks Combination S 1 S 2 S 3 S 4 S 5 S 6 S 7 S 8 S 9 S The new MMKP test set generated above yields a diverse correlation range as shown in Figures 5.2 and 5.3. The problems have a correlation range varying from ( 0.9, 0.9). The analytical MMKP problem set is used to study the effect of problem structure characteristics on the performance of various legacy heuristics; the set provides the ability to examine the effects of correlation structure, varied right-hand side levels, and the interaction of the two characteristics Competitive MMKP Test Problem Generation The base MDKP problem sets were used to generate competitive problem test sets. Rather than systematically varying constraint slackness settings, the slackness ratios are generated uniformly between (0.6, 0.9). For a tighter distribution (0.3, 0.7), the constraints were very tight yielding no solution to the problems. 95

112 Table 5.3: Coded Combinations of Slackness Settings for 25 Knapsacks Combination S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 S21 S22 S23 S24 S

113 Table 5.4: Correlation Between Objective and Constraint Coefficients Analysis of Available MMKP Test Problems Problem File min max (m,n,l) I (5,5,5) I (10,5,5) I (15,10,10) I (20,10,10) I (25,10,10) I (30,10,10) I (100,10,10) I (150,10,10) I (200,10,10) I (250,10,10) I (300,10,10) I (350,10,10) I (400,10,10) (m,n,l) represents (classes, items, resources) in problems Analytical MMKP Test Sets Versus Available MMKP Test Set This section compares the problem structure of the available MMKP test set with the new MMKP test set. Khan s (Khan et al. 2002) available test problems available at Hifi (2006) have 13 single test instances labeled I01 to I13 while the new MMKP test set generated has 9 files of 30 problems each. Correlation structure in Khan s 13 available test problems does not show much variation while the base test sets were generated by varying the correlation between ( 0.9, 0.9). Tables 5.4 and 5.5, respectively, show the variation in the correlation structure of the available MMKP test sets and the base MMKP test sets. 97

114 Table 5.5: Correlation Between Objective and Constraint Coefficients Analysis of New MMKP Test Sets Problem File min max (m,n,l) MMKP (5,10,5) MMKP (10,10,5) MMKP (25,10,5) MMKP (5,10,10) MMKP (10,10,10) MMKP (25,10,10) MMKP (5,10,25) MMKP (10,10,25) MMKP (25,10,25) (m,n,l) represents (classes, items, resources) in problems Figure 5.1 graphically shows the correlation range in Khan s 13 test problem instances. Figure 5.2 shows the variation of the correlation values for the base MMKP problem set with 30 problem instances generated with 5 classes, 10 items, and 5 knapsacks. The correlation structure for the other base MMKP problems are in Appendix A. Figure 5.3 plots the correlation structure for the entire set of the MMKP test problem instances. There is no variation in the right-hand side values of the problem constraints in Khan s 13 available test problems; the right-hand side values are the same for all the knapsack constraints in every problem. The new MMKP test problem instances are generated using Equation (5.1). Each of the knapsack constraints has varied capacity. Tables 5.6 and 5.7, respectively, show the right-hand side variation of the available MMKP test sets and the analytical MMKP test sets. Figures 5.4 and 5.5 graphically show the right-hand side range of Khan s 13 test problem instances and the newly generated MMKP test problem instances, respectively. An adequate test set should provide a wide range of test problem instances. Figures 5.1 and 98

115 Figure 5.1: Range of Correlation Values Between Objective Function and Constraint Coefficients for Available MMKP Test Problems Figure 5.2: Range of Correlation Values Between Objective Function and Constraint Coefficients for the generated MMKP Test Problem with 5 classes, 10 items, 5 knapsacks 99

116 Figure 5.3: Range of Correlation Values Between Objective Function and Constraint Coefficients for New MMKP Test Sets Table 5.6: Right-Hand Side Analysis of the Knapsack Constraints in Available MMKP Test Problems Problem File min max (m,n,l) I (5,5,5) I (10,5,5) I (15,10,10) I (20,10,10) I (25,10,10) I (30,10,10) I (100,10,10) I (150,10,10) I (200,10,10) I (250,10,10) I (300,10,10) I (350,10,10) I (400,10,10) (m,n,l) represents (classes, items, resources) in problems 100

117 Table 5.7: Right-Hand Side Analysis of the Knapsack Constraints of Analytical MMKP Test Sets Problem File min max (m,n,l) MMKP (5,10,5) MMKP (10,10,5) MMKP (25,10,5) MMKP (5,10,10) MMKP (10,10,10) MMKP (25,10,10) MMKP (5,10,25) MMKP (10,10,25) MMKP (25,10,25) (m,n,l) represents (classes, items, resources) in problems Figure 5.4: Range of the Right-Hand Side Values of the Knapsack Constraints of Available MMKP Test Problems 101

118 Figure 5.5: Range of the Right-Hand Side Values of the Knapsack Constraints of Analytical MMKP Test Sets 5.3 indicate that the available test problems have too narrow a range of correlation structure while the new test set has a wide range of correlation values. Figures 5.4 and 5.5 show that the available test problems have similar values of the right-hand sides for the knapsack constraints while the new test set varies the range of the right-hand side values. When test set instances do not provide enough diversity, empirically-based insights are not likely to generalize to real-world problem instances. 5.3 Empirical Analyses of MMKP Heuristics on Available Test Problems The legacy heuristic procedures proposed by Moser et al. (MOSER, 1997), Khan et al. (HEU, 2002), and Hifi et al. (CPCCP and DerAlgo, 2004) for the MMKP were programmed in Java and tested on the 13 test problems developed by Khan et al. (2002). The reported results, by the corresponding authors, are summarized in Table

119 Despite diligent efforts, the implemented versions of the legacy heuristics sometimes failed to meet reported results. The implemented versions faithfully capture the algorithms as published (via pseudo code). Attempts to obtain author versions of the codes were unsuccessful. Subsequent comparative tests involve the implemented versions since new test problems are employed. The optimal results obtained by programming and testing the respective heuristics on the 13 available test problems and the corresponding elapsed times in milliseconds are summarized in Table 5.9. The results indicate that exact optimal solutions could not be found for larger test problems I07 through I13. CPCCP and DerAlgo found solutions for all the 13 available MMKP test problem instances. The solution quality of CPCCP was close to that of DerAlgo while the computational time for CPCCP was the least amongst all the heuristics. MOSER and HEU often failed to find a solution. Tables 5.10 and 5.11 summarize competitive performance and the percentage relative error of the legacy heuristics on Khan s 13 test problems. DerAlgo performs the best the most number of times. CPCCP and DerAlgo have better performance than the other heuristics. MOSER and HEU often failed to find a feasible solution. As the size of the problem increases, the percentage relative error by CPCCP and DerAlgo decreases. Considering just the number of problems solved, HEU has the least average percentage relative error. Twenty-four test problem instances were generated using Khan s test problem generator varying the number of classes from 5 to 40; number of items from 5, 7, 9; and number of knapsack set to 5. The legacy MMKP heuristic methods were tested on these generated test problems. The results of the heuristic performance on these test problem instances are tabulated in Table Tables 5.13 and 5.14 summarize competitive performance and the percentage relative error of the legacy heuristics on these test problems. These results indicate that HEU compares favorably with the other heuristics. CPCCP and DerAlgo perform better than MOSER while CPCCP is better than DerAlgo. CPCCP is the fastest heuristic while HEU and MOSER are slower. The average percentage relative error for MOSER is higher than other heuristics and the MOSER and HEU 103

120 Table 5.8: Reported Results of the Legacy MMKP Heuristics on the 13 Available Test Problems Problem File Exact MOSER HEU CPCCP DerAlgo I I I I I I I * I * I * I * I * I * I * * LP Bound 1 Hifi et al. (2004) report ; is solution found 104

121 Table 5.9: Obtained Results of the Legacy MMKP Heuristics on the 13 Available Test Problems Problem File Exact MOSER Time HEU Time CPCCP Time DerAlgo Time I I I I I I I * I * I * I * I * I * I * Solution not found * LP Bound Time in ms 105

122 Table 5.10: Performance Summary of the Legacy Heuristics on Khan s 13 Available Test Problems Times Times Number of times Heuristic better than other approaches Heuristic Best Optimal MOSER HEU CPCCP DerAlgo MOSER HEU CPCCP DerAlgo total problems 106

123 Table 5.11: Percentage Relative Error of Legacy Heuristics on Khan s 13 Available Test Problems Problem File MOSER HEU CPCCP DerAlgo I I I I I I I I I I I I I Total Average Relative Error cannot be calculated since no solution found 107

124 Table 5.12: Results of the Legacy MMKP Heuristics on the Correlated Test Problems Generated using Khan s Test Problem Generator Prob No (m,n,l) Exact MOSER Time HEU Time CPCCP Time DerAlgo Time 1 (5,5,5) (5,7,5) (5,9,5) (10,5,5) (10,7,5) (10,9,5) (15,5,5) (15,7,5) (15,9,5) (20,5,5) (20,7,5) (20,9,5) (25,5,5) (25,7,5) (25,9,5) (30,5,5) (30,7,5) (30,9,5) (35,5,5) (35,7,5) (35,9,5) (40,5,5) (40,7,5) (40,9,5) Solution not found (m,n,l) represents (classes, items, resources) in problems Time in ms 108

125 heuristics often fail to find feasible solutions. The results are not comparable to results obtained on Khan s available test problems. 5.4 Empirical Analyses of MMKP Heuristics on New MMKP Test Problem Set The legacy heuristic procedures proposed by Moser et al. (MOSER, 1997), Khan et al. (HEU, 2002), and Hifi et al. (CPCCP, 2004) for the MMKP were then tested on the new MMKP test problem set. The purpose of this research is to gain insights on the effects of problem characteristics on heuristic performance. The heuristics are compared based on number of problems solved to optimal by each of the heuristics and the number of times each heuristic is the best performer. Any ties between heuristics are excluded. The DerAlgo heuristic is no longer considered based on concerns regarding the correctness of the implementation created based on the published literature Analyses based on Constraint Right-Hand Side Setting MMKP with 5 knapsacks The legacy heuristic performance on the MMKP test sets with 5 classes, 10 items, 5 knapsacks is summarized in Table The results show that the number of problems solved to optimal by all the heuristics is high for the problems with loose constraints because the problems are easy to solve. For loose constraints, the number of problems solved to optimal is the same for MOSER and CPCCP. When multiple constraints are tight, number of problems solved to optimal by MOSER is higher than HEU and CPCCP. When all the constraints are tight, CPCCP has the higher number of problems solved to optimal. The number of problems solved to optimal by HEU is low for all the constraint combinations. The number of problems solved to optimal reduces as the problem constraints get tighter. For the number of times the heuristics perform the best, HEU is outperformed with MOSER and CPCCP being preferred as more of the constraints get tighter. 109

126 Table 5.13: Performance Summary of the Legacy Heuristics on Extra Generated Test Problems Times Times Number of times Heuristic better than other approaches Heuristic Best Optimal MOSER HEU CPCCP DerAlgo MOSER HEU CPCCP DerAlgo total problems 110

127 Table 5.14: Percentage Relative Error of Legacy Heuristics on Extra Generated Test Problems Prob No MOSER HEU CPCCP DerAlgo Total Average Relative Error cannot be calculated since no solution found 111

128 Table 5.16 contains the results for the heuristic performance on the problems with 10 classes, 10 items, and 5 knapsacks. All three heuristics do well on problems with loose constraints. The results in Table 5.16 mirror the results in Table 5.15 although the MOSER approach appears to have the advantage in providing the best results among the three heuristics. The results for the number of problems solved to optimal and number of times best for the problem set with 25 classes, 10 items, and 5 knapsacks are reported in Table As before, all heuristics do well on problems with loose constraints while MOSER and CPCCP are equally adept on the more difficult problems. MMKP with 10 knapsacks Table 5.18 reports results on problems with 5 classes, 10 items, and 10 knapsacks. Tables 5.19 and 5.20 report results on problems with 10 classes and 25 classes, respectively. All three heuristics do well on problems with loose constraints although the HEU performance is degraded when compared to the 5 knapsack problems. In terms of returning a best solution, MOSER and CPCCP appear quite comparable until the 25 class problem set (Table 5.20) where the MOSER heuristic seems to have a definite advantage. For these larger problems, HEU does not compete well. MMKP with 25 knapsacks Table 5.21 reports results on problems with 5 classes, 10 items, and 25 knapsacks. Tables 5.22 and 5.23 report results on problems with 10 classes and 25 classes, respectively. None of the heuristics do particularly well in terms of finding the optimal solution although each heuristic finds more optimal solutions as the number of classes increase. An interesting aspect of these results involve the improved performance of CPCCP relative to MOSER on the problems with fewer classes (5 and 10 classes). For the larger classes, Table 5.23, results are mixed in terms of whether MOSER or CPCCP is preferred. 112

129 Table 5.15: Number of times best heuristic for each of the Right-Hand Side Combinations for 5 classes, 10 items, 5 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP Table 5.16: Number of times best heuristic for each of the Right-Hand Side Combinations for 10 classes, 10 items, 5 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP

130 Table 5.17: Number of times best heuristic for each of the Right-Hand Side Combinations for 25 classes, 10 items, 5 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP Table 5.18: Number of times best heuristic for each of the Right-Hand Side Combinations for 5 classes, 10 items, 10 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP

131 Table 5.19: Number of times best heuristic for each of the Right-Hand Side Combinations for 10 classes, 10 items, 10 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP

132 Table 5.20: Number of times best heuristic for each of the Right-Hand Side Combinations for 25 classes, 10 items, 10 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP

133 Table 5.21: Number of times best heuristic for each of the Right-Hand Side Combinations for 5 classes, 10 items, 25 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP

134 Table 5.22: Number of times best heuristic for each of the Right-Hand Side Combinations for 10 classes, 10 items, 25 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP

135 Table 5.23: Number of times best heuristic for each of the Right-Hand Side Combinations for 25 classes, 10 items, 25 knapsacks No. problems solved to optimal No. of times best Combination MOSER HEU CPCCP MOSER HEU CPCCP

136 5.4.2 Analyses based on Correlation Structure This section studies how the correlation structure of the problems affects heuristic performance. To study the effect of the correlation structure on the solution returned by the heuristic, a regression analysis was carried out on the analytical test problems generated and the solution obtained by each of the heuristics. The hypothesis is: H0: Correlation structure has no effect on the heuristic solution. H1: Correlation structure has an effect on the heuristic solution. The test statistic was calculated using Equation (5.2) F = MSR MSE (5.2) The decision rules used are in Equations (5.3) and (5.4). If F F (1 α; p 1, n p), conclude H0 (5.3) If F > F (1 α; p 1, n p), conclude H1 (5.4) where the level of significance α used is 0.1, p 1 is the numerator degrees of freedom, and n p is the denominator degrees of freedom. The legacy heuristic algorithms were tested on the new MMKP test problems. All significance levels used are for each individual test. Overall Regression Analysis The regression analysis involved all the design combinations for each of the problem files. The regression model is made up of the predictor variables and the response variable. The predictor variables for this model were the correlation value between the objective and the constraint coefficients for each of the constraints, while the response variable was the solution obtained by each 120

137 Table 5.24: Hypothesis Test Results of the Influence of Correlation Structure on Heuristic Performance Problem File MOSER HEU CPCCP 5GP10IT5KP Reject H0 Reject H0 Reject H0 10GP10IT5KP Reject H0 Reject H0 Reject H0 25GP10IT5KP Reject H0 Reject H0 Reject H0 5GP10IT10KP Reject H0 Reject H0 Reject H0 10GP10IT10KP Reject H0 Reject H0 Reject H0 25GP10IT10KP Reject H0 Reject H0 Reject H0 5GP10IT25KP FTR H0 Reject H0 Reject H0 10GP10IT25KP FTR H0 FTR H0 Reject H0 25GP10IT25KP Reject H0 Reject H0 Reject H0 FTR = Fail to Reject of the heuristic approaches. The results for the overall regression analysis of the heuristic solution performance versus the correlation structure are tabulated in Table The results indicate that the correlation structure does affect heuristic performance for all the problems with 5 knapsacks and 10 knapsacks. As the size of the problems increase with 25 knapsacks, the correlation effect reduces. Regression Analysis for each of the problem types Results using each of the new MMKP test problem sets were also analyzed by the correlation structure combination setting. The hypothesis for each of the problem types is: H0: Correlation structure has no effect on the heuristic solution. H1: Correlation structure has an effect on the heuristic solution. The test statistic was calculated using Equation (5.5) 121

138 F = MSR MSE (5.5) The decision rules used are in Equations (5.6) and (5.7). If F F (1 α; p 1, n p), conclude H0 (5.6) If F > F (1 α; p 1, n p), conclude H1 (5.7) where the level of significance α used is 0.1, p 1 is the numerator degrees of freedom, and n p is the denominator degrees of freedom. The predictor variables for this model were the correlation value between the objective and the constraint coefficients for each of the constraints, while the response variable was the solution obtained by each of the heuristic approaches. Table 5.25 contains the results using the new MMKP test sets with 5 knapsacks by right-hand side combination settings. The tabulated results in Table 5.25 indicate that for Combination 1 with 5 knapsacks where the constraints are loose, and the problem is easy to solve, correlation structure does not have any effect on the heuristic performance of MOSER, HEU, and CPCCP for 5 and 25 classes. As multiple constraints get tighter, the correlation structure affects heuristic performance indicating an interaction effect between constraint settings and correlation. Table 5.26 summarizes the analyses on the effect of the correlation structure on the heuristic performance based on the constraint tightness settings for the problems with 10 knapsacks. The results indicate that for Combination 1 where the problems are easy to solve, correlation structure does not effect heuristic performance. As the constraints get tighter, MOSER and HEU s performance is not as affected by the correlation structure while CPCCP s performance is affected. These results indicate that problem structure affects heuristic performance. The regression analysis for the correlation structure on the legacy heuristic performance based on the constraint tightness settings for the problems with 25 knapsacks and varying classes is tabu- 122

139 Table 5.25: Hypothesis Test Results of the Influence of the Correlation Structure on Heuristic Performance based on Constraint Tightness for MMKP problems with 5 knapsacks, 10 items, and varied classes of 5, 10, and 25 5GP10IT5KP 10GP10IT5KP 25GP10IT5KP Combination MOSER HEU CPCCP MOSER HEU CPCCP MOSER HEU CPCCP 1 FTR H0 FTR H0 FTR H0 FTR H0 Reject H0 Reject H0 FTR H0 FTR H0 FTR H0 2 Reject H0 Reject H0 Reject H0 FTR H0 Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 3 Reject H0 Reject H0 Reject H0 FTR H0 Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 4 Reject H0 FTR H0 Reject H0 Reject H0 FTR H0 FTR H0 Reject H0 Reject H0 Reject H0 5 Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 6 Reject H0 - Reject H0 Reject H0 Reject H0 Reject H0 Reject H0 FTR H0 Reject H0 FTR = Fail to Reject - Solution not found 123

140 Table 5.26: Hypothesis Test Results of the Influence of the Correlation Structure on Heuristic Performance based on Constraint Tightness for MMKP problems with 10 knapsacks, 10 items, and varied classes of 5, 10, and 25 5GP10IT10KP 10GP10IT10KP 25GP10IT10KP Combination MOSER HEU CPCCP MOSER HEU CPCCP MOSER HEU CPCCP 1 FTR H0 Reject H0 FTR H0 FTR H0 FTR H0 FTR H0 FTR H0 FTR H0 FTR H0 2 FTR H0 FTR H0 Reject H0 Reject H0 Reject H0 FTR H0 FTR H0 Reject H0 FTR H0 3 FTR H0 FTR H0 Reject H0 FTR H0 Reject H0 FTR H0 FTR H0 Reject H0 FTR H0 4 FTR H0 FTR H0 Reject H0 Reject H0 Reject H0 FTR H0 FTR H0 FTR H0 FTR H0 5 FTR H0 FTR H0 FTR H0 FTR H0 Reject H0 Reject H0 FTR H0 FTR H0 Reject H0 6 FTR H0 FTR H0 Reject H0 FTR H0 Reject H0 Reject H0 FTR H0 FTR H0 Reject H0 7 FTR H0 FTR H0 Reject H0 FTR H0 FTR H0 Reject H0 Reject H0 FTR H0 Reject H0 8 FTR H0 FTR H0 Reject H0 FTR H0 FTR H0 Reject H0 Reject H0 - Reject H0 9 FTR H0 - Reject H0 FTR H0 - Reject H0 FTR H0 - Reject H0 10 FTR H0 - Reject H0 FTR H0 - Reject H0 FTR H0 - FTR H0 11 FTR H0 - Reject H0 FTR H0 - Reject H0 FTR H0 - Reject H0 FTR = Fail to Reject - Solution not found 124

141 lated in Table The results indicate a reduction in the problem structure effects on the heuristic performance for larger problems with 25 knapsacks. 5.5 Summary This chapter examined the test problem generation methods for the available MMKP test sets and our new MMKP test sets. The new MMKP test sets were generated by varying problem characteristics such as the number of classes, number of items, number of knapsacks, constraint right-hand side setting, and correlation structure. This chapter then examined performance of each of three MMKP legacy heuristics - MOSER, HEU, CPCCP, based on problem structure - constraint right-hand side setting and correlation structure. A total of 180 problems were generated for the problem sets with 5 knapsacks, a total of 330 problems were generated for the problem sets with 10 knapsacks, and a total of 780 problems were generated for the problem sets with 25 knapsacks. A total of 3870 MMKP problem instances were generated. The research considered which of the heuristics yielded the best solution under varying conditions. The heuristics were empirically analyzed based on the constraint right-hand side settings and the correlation structure. The empirical analyses based on the constraint right-hand side settings indicate that the performance of MOSER and CPCCP is comparable when all the problem constraints are loose. As multiple constraints get tight MOSER generally outperforms CPCCP and HEU. For the problems with all tight constraints, CPCCP is the clear winner, meaning CPCCP is preferred when all constraints are equivalently defined. Regression analysis was used to study the effect of the correlation structure on the heuristic performance. The results indicated that there is an effect on the heuristic performance due to the problem correlation structure. Caveats to this conclusion include little effect while problem constraints are loose and a diminished interaction between problem correlation structure and constraint tightness as the number of knapsacks increase. 125

142 Table 5.27: Hypothesis Test Results of the Influence of the Correlation Structure on Heuristic Performance based on Constraint Tightness for MMKP problems with 25 knapsacks, 10 items, and varied classes of 5, 10, and 25 5GP10IT25KP 10GP10IT25KP 25GP10IT25KP Combination MOSER HEU CPCCP MOSER HEU CPCCP MOSER HEU CPCCP 1 FTR H0 FTR H0 Reject H0 FTR H0 FTR H0 FTR H0 FTR H0 FTR H0 Reject H0 2 FTR H0 FTR H0 FTR H0 FTR H0 FTR H0 Reject H0 Reject H0 FTR H0 Reject H0 3 FTR H0 FTR H0 FTR H0 Reject H0 FTR H0 Reject H0 Reject H0 FTR H0 FTR H0 4 FTR H0 FTR H0 Reject H0 Reject H0 FTR H0 Reject H0 FTR H0 FTR H0 FTR H0 5 FTR H0 Reject H0 FTR H0 FTR H0 - Reject H0 FTR H0 FTR H0 FTR H0 6 FTR H0 Reject H0 FTR H0 FTR H0 - Reject H0 Reject H0 FTR H0 FTR H0 7 FTR H0 Reject H0 Reject H0 FTR H0 - FTR H0 FTR H0 FTR H0 FTR H0 8 Reject H0 Reject H0 FTR H0 FTR H0 - FTR H0 Reject H0 FTR H0 FTR H0 9 Reject H0 Reject H0 FTR H0 FTR H0 - FTR H0 FTR H0 FTR H0 FTR H0 10 FTR H0 Reject H0 FTR H0 FTR H0 - FTR H0 FTR H0 FTR H0 FTR H0 11 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 FTR H0 FTR H0 12 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 FTR H0 FTR H0 13 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 FTR H0 FTR H0 14 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 FTR H0 FTR H0 15 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 - FTR H0 16 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 - FTR H0 17 FTR H0 - FTR H0 Reject H0 - FTR H0 FTR H0 - FTR H0 18 FTR H0 - Reject H0 FTR H0 - FTR H0 FTR H0 - FTR H0 19 FTR H0 - Reject H0 FTR H0 - FTR H0 FTR H0 - FTR H0 20 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 - FTR H FTR H0 Reject H0 - FTR H0 FTR H0 - FTR H0 22 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 - FTR H0 23 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 - FTR H0 24 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 - FTR H FTR H0 - - FTR H0 FTR H0 - FTR H0 26 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR H0 - FTR H0 FTR = Fail to Reject - Solution not found 126

143 6. New Greedy Heuristics for the MMKP 6.1 Introduction This chapter presents new heuristic approaches for solving the MMKP based on the insights gained from the previous empirical study. The first heuristic introduced is a TYPE-based heuristic. TYPE pre-processes a problem and uses that problem-specific knowledge to obtain computational efficiencies. The second heuristic is developed by solving the relaxed MMKP as a modified MDKP and using a greedy heuristic to obtain a good initial solution. This initial solution is improved by a local improvement phase. The third heuristic extends the second heuristic adding a more aggressive local improvement phase. 6.2 A TYPE-based Heuristic for the MMKP A TYPE-based heuristic is a heuristic designed to pick a likely best performer among a suite of heuristics by pre-processing a problem. The pre-processing of a problem involves determining problem characteristics such as constraint tightness, problem correlation structure, and problem size, and then selecting a heuristic approach that is likely to produce a best solution from a given collection of heuristic approaches. Such an approach was suggested by Loulou and Michaelides (1979) and first realized by Cho et al. (2003b) for the MDKP. In this research, three legacy heuristic approaches, MOSER, HEU, and CPCCP are used as the suite of MMKP legacy heuristics. Based on the previous empirical study in Chapter 5 among three 127

144 Table 6.1: MMKP Problems with 5 Knapsacks, Different Right-Hand Side Combinations, and Best Heuristic Problem File Combination 5GP10IT5KP 10GP10IT5KP 25GP10IT5KP 1 HEU(CPCCP) CPCCP CPCCP 2 MOSER MOSER HEU(MOSER) 3 MOSER MOSER MOSER 4 MOSER MOSER MOSER 5 MOSER/CPCCP MOSER MOSER 6 CPCCP MOSER CPCCP different MMKP heuristics - MOSER, HEU, and CPCCP, the best performer based on the type of the problem and the constraint right-hand side (slackness combination) was determined. From the empirical study for the problems with 5 knapsacks, MOSER is the best performer when multiple constraints are tight. When all the constraints are tight, CPCCP performs the best. MOSER and CPCCP perform the same for loose constraints. For the problems with 10 knapsacks, MOSER is the best for the problems having less than eight tight constraints while CPCCP performs the best when eight or more of the constraints are tight. In the case of problems with loose constraints, both MOSER and CPCCP perform the same. Figure 6.1 flowcharts the TYPE-based heuristic defined and implemented. Tables 6.1 and 6.2 summarize the best heuristic performance for different settings of problem type and right-hand side combinations for problems with 5 and 10 knapsacks with varying number of classes, respectively. The results in these tables are used to specify a TYPE heuristic, a heuristic that uses problem characteristics to pick a specific heuristic. Figure 6.1 details the TYPE heuristic. Analysis of the TYPE-based Heuristic Results for the MMKP TYPE-based heuristic was tested on the analytical MMKP problems discussed in Section

145 Table 6.2: MMKP Problems with 10 Knapsacks, Different Right-Hand Side Combinations, and Best Heuristic Problem File Combination 5GP10IT10KP 10GP10IT10KP 25GP10IT10KP 1 CPCCP HEU(MOSER/CPCCP) HEU(MOSER/CPCCP) 2 CPCCP MOSER HEU(MOSER) 3 MOSER MOSER MOSER 4 MOSER MOSER MOSER 5 CPCCP MOSER MOSER 6 MOSER MOSER MOSER 7 MOSER MOSER MOSER 8 MOSER MOSER/CPCCP MOSER 9 CPCCP CPCCP MOSER 10 CPCCP MOSER MOSER 11 CPCCP MOSER/CPCCP CPCCP Figure 6.1: Flowchart for TYPE-based Heuristic 129

146 Table 6.3: Number of Times Equal to Best of the Legacy Heuristic Solutions according to Problem Type Problem File MOSER HEU CPCCP TYPE MMKP MMKP MMKP MMKP MMKP MMKP Table 6.3 summarizes the best heuristics under constraint tightness and the type of the problem. These results are graphically plotted in Figure 6.2. The data for the TYPE-based heuristic performance includes ties to demonstrate its performance compared to the legacy MMKP heuristics. The TYPE-based MMKP heuristic has comparable performance to the best legacy heuristic using some computational inexpensive pre-processing to select a heuristic versus running each heuristic and returning the best solution. 6.3 New Greedy Heuristic Version 1 (CH1) A new greedy heuristic for solving the MMKP is developed. This heuristic first solves a relaxed version of the MMKP to obtain a good initial feasible solution. A greedy heuristic NG V3 (Cho 2005) is used to solve the MDKP problem. A final solution is obtained by using a local improvement phase to improve the initial solution. The subsequent subsections explain the NG V3 heuristic and the implementation of the new greedy heuristic for the MMKP (CH1) NG V3 Heuristic (Cho 2005) 130

147 Figure 6.2: Comparison of TYPE-based Heuristic according to Problem Type Cho (2005) conducted an empirical study for the MDKP and developed three new improved gradient heuristics that exploit the insights of his empirical MDKP study. His NG V1 improves the solution trajectory through the feasible region. NG V2 modifies a delayed constraint weighting scheme to be suitable for any constraint slackness setting. NG V3 is the heuristic combining aspects of NG V1 and NG V2. NG V3 improves the effective gradient function using a lognormal distribution to improve its response to various combinations of constraint slackness setting and correlation structures. NG V3 analyzes the problem characteristics and pre-weights the dominant constraint to use resources more effectively starting at the initial iteration. Computational results comparing his new gradient heuristics with a variety of legacy MDKP heuristics showed that NG V3 outperformed all the legacy heuristics considered (Cho 2005). The flowchart for the NG V3 (Cho 2005) heuristic is shown in Figure 6.3. Cho (2005) computes the pre-weighting parameters for each of the knapsack constraints by Equation (6.1). P rew eight(i) = exp(ρ CA i) exp(1/r i ) (6.1) 131