HETEROGENEOUS COMPUTING

Transcription

1 HETEROGENEOUS COMPUTING Shoukat Ali, Tracy D. Braun, Howard Jay Siegel, and Anthony A. Maciejewski School of Electrical and Computer Engineering, Purdue University Heterogeneous computing is a set of techniques enabling the use of diverse computational capabilities for the execution of a meta-task [2, 4, 7]. A meta-task is an arbitrary collection of independent (non-communicating) tasks with a variety of computational needs, which are to be executed during a given interval of time (e.g., a day). Some tasks may be decomposable into one or more communicating subtasks (subtasks may, in turn, have diverse computational needs). There are many types of heterogeneous computing systems [2]. This article focuses on mixed-machine systems, where a heterogeneous suite of independent machines is interconnected by high-speed links to function as a meta-computer (see Meta-Computer) or as part of a computational grid (see Computational Grid) [3]. The user of a heterogeneous suite has the illusion of working on a single virtual machine. Research in the field of heterogeneous computing is motivated by the fact that high performance machines vary in capability and, hence, suitability for different types of tasks and subtasks. Examples of such machine architectures include large distributed shared memory machines (e.g., an SGI 2800), distributed memory multiprocessors (e.g., an IBM SP2), and small shared memory machines (e.g., a Sun Enterprise 3000 Server). Furthermore, two implementations of a given machine type may vary in CPU speed, cache memory size and structure, I/O bandwidth, etc. With the recent advances in high-speed digital communications, it has become possible to use collections of different machines in concert to execute large meta-tasks whose tasks and subtasks have diverse computational needs. The goal of heterogeneous computing is to assign these tasks and subtasks to machines and schedule their execution to optimize some performance measure. This measure may be as simple as the execution time of the meta-task. The measure may be more complex and be a mathematical function of various factors such as the weighted priorities of tasks, deadlines for task execution, security requirements, and quality of service (QoS) needs (see QoS). The process of assigning (matching) tasks/subtasks to machines and scheduling their execution is called mapping. A hypothetical example task with four subtasks that are best suited for different machine architectures is shown in Figure 1. The example task executes for 100 time units on a typical workstation. The task consists of four subtasks: the first (S1) is best suited to execute on a large cluster of PCs (e.g., a Beowulf Cluster), the second (S2) is best suited to execute on a distributed memory multiprocessor, the third (S3) is best suited to execute on a distributed shared memory machine, and the fourth (S4) is best suited to execute on a small shared memory machine. Executing the whole task on a large cluster may improve the execution time of the first subtask from 25 to 0.3 time units, and those of the other subtasks to varying extents. The overall execution time improvement may only be about a factor of five because other subtasks are not well suited for execution on a cluster (e.g., due to the need for interprocessor communication). However, using four different machines that match the computational requirements for each of the individual subtasks can result in an overall execution time that is better than the execution time on the workstation by a factor of over 50. For communicating subtasks, inter-machine data transfers need to be performed when multiple machines are used. Hence, data transfer overhead has to be considered as part of the overall execution time on the heterogeneous computing suite whereas there is no such overhead when the entire task is executed on a single workstation. This is a simplified example. Actual tasks may consist of a large number of subtasks with a much more complex intersubtask communications structure. Also, the sharing of the machines by all the tasks in the meta-task must be considered when mapping. Mapping Finding a mapping for tasks that optimizes some performance measure is, in general, an NP-complete problem. For example, consider mapping 30 tasks onto five machines. This means that there are 5 30 possible mappings. Even if it took only one nanosecond to evaluate each mapping, an exhaustive search to find the best mapping would require 5 30 nanoseconds > 1000 years! Therefore, it is necessary to have heuristics to find near-optimal mappings without using an exhaustive search. Factors that impact mapping decisions include: (1) match of the task computational requirements to the machine capabilities, (2) overhead for the inter-machine communication of code and data (initial and generated), (3) expected machine load and network congestion, and (4) inter-subtask precedence constraints. There are many different types of heuristics for mapping tasks to the machines in a heterogeneous computing suite. In static mapping heuristics [1], the mapping decisions are made off-line before the execution of the meta-task. A static

2 mapping heuristic is employed if (1) the tasks that comprise the meta-task are known a priori, (2) predictions about the available heterogeneous computing resources are likely to be accurate, and (3) the estimated expected execution time of each task on each machine in the suite is known reasonably accurately. Static mapping heuristics can be used for planning the next day s work on a heterogeneous computing system. execution on a baseline workstation S1 S2 S3 S units of time on a baseline workstation execution on a cluster five times faster than baseline execution on a heterogeneous suite times faster than baseline Figure 1. Hypothetical example (based on [4]) of the advantage of using a heterogeneous computing suite of machines. The number underneath each bar indicates execution time. For the suite, each subtask execution time includes the overhead to receive data. Not drawn to scale. In dynamic mapping heuristics [6], the mapping decisions are made on-line during the execution of the meta-task. Dynamic approaches to mapping are needed if any of the following are unpredictable: (1) arrival times of the tasks, (2) machines available in the heterogeneous computing system (some machines in the suite may go off-line and new machines may come on-line), and (3) expected execution times of the tasks on the machines. While a static mapper considers the entire meta-task to be executed (e.g., the next day) when making decisions, a dynamic mapper has only information about tasks that have already arrived for execution. Furthermore, because a dynamic mapper operates on-line, it must make decisions much faster than an off-line static mapper. Consequently, dynamic mapping heuristics often use feedback from the heterogeneous computing system (while tasks are executing) to improve any bad mapping decisions. A semi-static mapping heuristic [8] can be used for an iterative task whose subtask execution times will change from iteration to iteration based on the input data. A semi-static methodology observes, from one iteration to another, the effects of the changing characteristics of the task's input data, called dynamic parameters, on the task's execution time. The off-line phase uses a static mapping algorithm to generate high quality mappings for a sampling of values for the dynamic parameters a priori. During the on-line phase, the actual dynamic parameters are observed and a new mapping for the subtasks may be selected from the precomputed off-line mappings. Automatic Heterogeneous Computing One of the long-term goals of heterogeneous computing research is to develop software environments that will automatically map and execute tasks expressed in a machine-independent high-level language. Such an environment will facilitate the use of heterogeneous computing suite by increasing portability, because the programmer need not be concerned with the composition of the heterogeneous computing suite, and increasing the possibility of deriving better mappings than the user can derive with ad hoc methods. Thus, it will improve the performance of and encourage the use of heterogeneous computing. While no such environment exists today, many researchers are working to develop one. A conceptual model for such an environment using a dedicated heterogeneous computing suite of machines is described in Figure 2, and consists of four stages. Stage 1 uses information about the type of tasks in the meta-task and machines in the heterogeneous computing suite to generate a set of parameters relevant to both the computational characteristics of tasks and the architectural features of machines. The system then derives categories for computational requirements and categories for machine capabilities from this set of parameters. Stage 2 consists of two components: task profiling and analytical benchmarking. Task profiling (see Task Profiling) decomposes each task of the meta-task into subtasks, where each subtask is computationally homogeneous. The computational requirements of each subtask are quantified by profiling the code and data. Analytical benchmarking

3 (see Analytical Benchmarking) quantifies how effectively each available machine in the suite performs on each type of computational requirement. The information available from stage 2 is used by stage 3 to derive the estimated execution time of each subtask on each machine in the heterogeneous computing suite, along with the associated inter-machine communication overheads. These statically derived results are then incorporated with initial values for machine ready times, intermachine network delays, and status parameters (e.g., machine/network faults) to perform the mapping of subtasks to machines based on a given performance metric. The result is an assignment of subtasks to machines and an execution schedule. The process as described corresponds to a static mapping. The subtasks are executed in stage 4. If dynamic mapping is employed, the subtask completion times and loading/status of the machines/network are monitored (shown in dashed lines in Figure 2). The monitoring process is necessary because the actual computation times and data transfer times may be input-data dependent and deviate considerably from the static estimates. This information may be used to re-invoke the mapping of stage 3 to improve the machine assignment and execution schedule. STAGE 1 applications generation of parameters relevant to both applications and machines machines in suite categories for computational needs categories for machine capabilities 2 task profiling for given applications analytical benchmarking for machines decomposition into subtasks, characteristics of each subtask initial status of machines, network; performance measure machine characteristics, inter-machine communication overhead 3 mapping matching of tasks to machines, execution schedule current status of machines and network; value of performance measure 4 execution monitor Figure 2. Model for integrating the software support needed for automating the use of heterogeneous computing systems (based on [7]). Ovals indicate information and rectangles indicate action.

4 Environments and Applications Examples of heterogeneous computing environments are: (1) the Purdue University Network Computing Hubs, a wide area network computing system which can be used to run a selection of software tools via a World Wide Web browser [5]; (2) NetSolve, a client-server system with geographically distributed servers that can be accessed from a variety of interfaces, including MATLAB, shell scripts, C, and FORTRAN [3]; and (3) the Globus meta-computing infrastructure toolkit, a set of low-level mechanisms that can be built upon to develop higher level heterogeneous computing services [3]. Example applications that have demonstrated the usefulness of heterogeneous computing include: (1) a threedimensional simulation of mixing and turbulent convection at the Minnesota Supercomputer Center [7]; (2) the shipboard anti-air warfare program (HiPer-D) used at the Naval Surface Warfare Center for threat detection, target engagement, and missile guidance; and (3) a simulation of colliding galaxies performed by solving large n-body dynamics problems and large gas dynamics problems at the National Center for Supercomputing Applications [7]. Open Problems in Heterogeneous Computing Heterogeneous computing is a relatively new research area for the computer field. Interest in such systems continues to grow both in the research community and in the user community. The realization of the automatic heterogeneous computing environment envisioned in Figure 2 requires further research in many areas. Machine-independent languages with user-specified directives are needed to (1) allow compilation of a given task into efficient code for any machine in the suite, (2) aid in decomposing tasks into subtasks, and (3) facilitate determination of subtask computational requirements. Moreover, methods must be refined for measuring the loading and status of the machines in the heterogeneous computing suite and the network, and for estimating the subtask completion times. Also, the uncertainty present in the estimated parameter values, such as subtask completion times, should be taken into consideration in determining the mappings. Other research areas are (1) developing communication protocols for reliable, low overhead data transmission over heterogeneous networks with given QoS requirements, (2) devising debugging tools that can be used transparently across the suite of machines, and (3) formulating algorithms for task migration between heterogeneous machines, using task migration for fault tolerance or load re-balancing. Acknowledgment: This work was supported by the DARPA/ITO Quorum Program through the Office of Naval Research under Grant No. N References [1] T. D. Braun, H. J. Siegel, N. Beck, L. L. Boloni, M. Maheswaran, A. I. Reuther, J. P. Robertson, M. D. Theys, B. Yao, D. Hensgen, and R. F. Freund, A Comparison of Eleven Static Heuristics for Mapping a Class of Independent Tasks onto Heterogeneous Distributed Computing Systems, Journal of Parallel and Distributed Computing, 61(6), , [2] M. M. Eshaghian (ed.), Heterogeneous Computing, Artech House, Norwood, MA, [3] I. Foster and C. Kesselman (eds.), The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, San Francisco, CA, [4] R. F. Freund and H. J. Siegel (guest eds.), Special Issue on Heterogeneous Processing, IEEE Computer, 26(6), [5] N. H. Kapadia and J. A. B. Fortes, PUNCH: An Architecture for Web-Enabled Wide-Area Network-Computing, Cluster Computing: The Journal of Networks, Software Tools and Applications, Special Issue on High Performance Distributed Computing, 2(2), , [6] M. Maheswaran, S. Ali, H. J. Siegel, D. Hensgen, and R. F. Freund, Dynamic Mapping of a Class of Independent Tasks onto Heterogeneous Computing Systems, Journal of Parallel and Distributed Computing, Special Issue on Software Support for Distributed Computing, 59(2), , [7] M. Maheswaran, T. D. Braun, and H. J. Siegel, Heterogeneous Distributed Computing, in Encyclopedia of Electrical and Electronics Engineering, Vol. 8, J. G. Webster, ed., John Wiley, New York, NY, 1999, pp [8] M. D. Theys, T. D. Braun, Y.-K. Kwok, H. J. Siegel, and A. A. Maciejewski, Mapping of Tasks onto Distributed Heterogeneous Computing Systems Using a Genetic Algorithm Approach, in Solutions to Parallel and Distributed Computing Problems: Lessons from Biological Sciences, A. Y. Zomaya, ed., John Wiley & Sons, New York, NY, 2001, pp

5 Cross Reference: Analytical Benchmarking see Heterogeneous Computing. Computational Grid see Heterogeneous Computing. Meta-Computer see Heterogeneous Computing. Meta-Task see Heterogeneous Computing. Task Profiling see Heterogeneous Computing. Dictionary Terms: Analytical Benchmarking Analytical benchmarking of a given computing machine provides a measure of performance of the machine on each of the different code types that may be present in a given source program. The performance of a particular code type on a specific kind of resource is a multi-variable function. Some of the variables of such a function may be: the quality of service requirements of the application program (e.g., data precision), the size of the data set to be processed, the algorithm to be applied, programmer and compiler efforts to optimize the program, and the operating system and architecture of the machine that will execute the specific code type. (See Computational Grid A developing area of research and technology seeking to connect regional and national computational resources in a transparent fashion, thus transforming any computer connected to the grid into part of a new class of supercomputer. The implied analogy is with an electric power grid. If access to advanced computational capabilities and accessories became as ubiquitous and dependable as an electric power grid, the impact on society would be dramatic. (See Meta-Computer A system framework that utilizes the resources of many different computers connected via a network to cooperate on solving a problem. In general, this allows the problem to be solved much more quickly than would be possible using a single computer. Meta-computers usually consist of heterogeneous, distributed elements, and operate in a coarsegrained fashion. A meta-computer would be a more localized component of a larger computational grid. (See QoS QoS (Quality of Service) is an aggregate function of many different system characteristics used to represent the overall performance of a system. The components in the function, and the computation of the function itself, vary widely (i.e., QoS means many different things to many different people). Sample components of a QoS measure could include task deadlines, data precision, image color range, video jitter, network bandwidth, bit error rate, and end-to-end latency. (See Task Profiling Task profiling of a given source program specifies types of computations that are present in the source program by decomposing it into code blocks based on computational requirements of the blocks. The set of computation types defined depends on the architectural features of the machines available for executing the source program or its subprograms, and on both the application task code and the types and sizes of data sets it is to process. (See