Computer Communications and Networks

Transcription

1 Computer Communications and Networks For other titles published in this series, go to

2 The Computer Communications and Networks series is a range of textbooks, monographs and handbooks. It sets out to provide students, researchers and non-specialists alike with a sure grounding in current knowledge, together with comprehensible access to the latest developments in computer communications and networking. Emphasis is placed on clear and explanatory styles that support a tutorial approach, so that even the most complex of topics is presented in a lucid and intelligible manner.

3 Maciej Drozdowski Scheduling for Parallel Processing 123

4 Maciej Drozdowski Poznan University of Technology Inst. Computing Science ul. Piotrowo Poznan Poland Maciej.Drozdowski@cs.put.poznan.pl Series Editor Professor A.J. Sammes, BSc, MPhil, PhD, FBCS, CEng Centre for Forensic Computing Cranfield University DCMT, Shrivenham Swindon SN6 8LA UK ISSN ISBN e-isbn DOI / Springer Dordrecht Heidelberg London New York British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Control Number: c Springer-Verlag London Limited 2009 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licenses issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. The use of registered names, trademarks, etc., in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Cover design: SPi Publisher Services Printed on acid-free paper Springer is part of Springer Science+Business Media (

5 Preface Overview and Goals This book is dedicated to scheduling for parallel processing. Presenting a research field as broad as this one poses considerable difficulties. Scheduling for parallel computing is an interdisciplinary subject joining many fields of science and technology. Thus, to understand the scheduling problems and the methods of solving them it is necessary to know the limitations in related areas. Another difficulty is that the subject of scheduling parallel computations is immense. Even simple search in bibliographical databases reveals thousands of publications on this topic. The diversity in understanding scheduling problems is so great that it seems impossible to juxtapose them in one scheduling taxonomy. Therefore, most of the papers on scheduling for parallel processing refer to one scheduling problem resulting from one way of perceiving the reality. Only a few publications attempt to arrange this field of knowledge systematically. In this book we will follow two guidelines. One guideline is a distinction between scheduling models which comprise a set of scheduling problems solved by dedicated algorithms. Thus, the aim of this book is to present scheduling models for parallel processing, problems defined on the grounds of certain scheduling models, and algorithms solving the scheduling problems. Most of the scheduling problems are combinatorial in nature. Therefore, the second guideline is the methodology of computational complexity theory. In this book we present four examples of scheduling models. We will go deep into the models, problems, and algorithms so that after acquiring some understanding of them we will attempt to draw conclusions on their mutual relationships. Hence, this book is also intended to give generalizations on scheduling models, their problems, and algorithms. Target Audiences This book may be helpful for researchers in parallel computing, operating systems, management science, and applied mathematics. We hope that this book may be also useful for lecturers and advanced students in the above fields. v

6 vi Preface A necessary prerequisite to reading this book is understanding basic concepts in computer hardware, programming, operating systems, parallel processing, and discrete mathematics. Knowledge of the notions from combinatorial optimization, computational complexity, and parallel application performance metrics would ease reading the book. Organization and Features The book is organized in the following way. Chapters 1, 2, and 3 set the stage for discussing scheduling models. Four examples of scheduling models are presented in Chaps. 4 to 7. Conclusions on applicability of scheduling models are drawn in Chap. 8. A more detailed description of the following chapters is given below. In Chap. 1 fundamental scheduling concepts which appear throughout the book are introduced. The relation between scheduling models, problems, algorithms, and schedules is clarified. In Chap. 2 selected notions related to graph theory are introduced to avoid ambiguity in further chapters. The methodology of computational complexity theory is outlined. It is a framework for scheduling as a combinatorial optimization problem. Finally, basic metrics of parallel application performance are introduced. Chapter 3 is dedicated to the technological aspects of scheduling for parallel processing. Scheduling determinants posed by hardware, programming environments, and widely understood operating systems are discussed. We examine the perception of and support for scheduling in parallel systems. This chapter shows how highly abstract scheduling policies are determined by the underlying hardware and software. Chapter 4 is dedicated to scheduling parallel applications on the grounds of the classic deterministic scheduling theory. We present the notions, concepts, and algorithms that are most immediately applicable in parallel processing. A reader not familiar with the scheduling theory may find here many concepts and definitions shared by other models in the book. The classic scheduling theory has developed in many directions which are applicable in parallel processing. Selected extensions of the classic theory are discussed. Chapter 5 examines the parallel task model. In the parallel task model a task may be executed by several processors simultaneously. The internal structure of the task is unknown to the scheduler. However, the scheduler may know some of the task requirements. Several types of parallel tasks are analyzed: rigid tasks which require a fixed number of processors, moldable tasks which may be executed on several alternative numbers of processors but the actual number of used processors is selected by the scheduler before starting the application, malleable tasks which may change the number of used processors during runtime, multiprocessor tasks which require a set of processors, and tasks requesting a certain processor interconnection shape (for instance mesh or hypercube). This chapter may be interesting for designers of operating systems and queuing systems for supercomputing facilities. Chapter 6 examines scheduling with communication delays. A parallel application is a collection of sequential communicating processes (tasks) here. If two tasks

7 Preface vii are executed on different processors, and the first one is producing some data for the second one, then a communication delay arises as a result of transferring data from one processor to the other. Scheduling algorithms must take into account costs of communication delays. The algorithms presented in this chapter may be interesting for programmers, developers of parallel applications, and designers of compilers. In Chap. 7 scheduling divisible loads is discussed. This model applies to scheduling parallel applications which can be divided into parts of arbitrary sizes, which in turn can be processed independently on remote computers. The assumption about arbitrary divisibility means in practice that the grains of parallelism are fine. Independent computing on remote computers means that there are no data dependencies between the grains of parallelism. This model has been applied to schedule parallel applications in various processor interconnection networks, and in systems with limited memory, to minimize the cost of parallel execution, and to analyze performance of parallel computer systems. The content of this chapter may be useful for developers of parallel applications and designers of parallel systems. Finally, in Chap. 8 we return to the initial division of the research area into domains of scheduling models. We study the relations between models, problems, and algorithms using the knowledge presented in the earlier chapters. This book presents ideas and concepts rather than proofs of the theorems. Moreover, we will only outline some methods which are too complex to be presented in fine detail. Still, we hope that the given description will be sufficient to grasp the idea of the presented algorithms, allowing the readers to code them if necessary. Many scheduling algorithms presented in this book are heuristics solving hard combinatorial problems. We do not compare the performance of the proposed heuristic methods, as it may be taken for granted that there are instances for which certain heuristics perform well. On the other hand, discussing generality of such results is beyond the scope of this book. The algorithms considered in this book are presented in the order they appeared in print. The index at the end of the book may be helpful in finding a particular algorithm. With the exception of a few symbols like the number of tasks n, number of processors m, schedule length C max, the notation is local for the chapters. The symbols which are frequently used are summarized in Appendix A. Acknowledgements Writing this book would not be possible without help from many people. I would like to thank my family for supporting me and giving courage to go through the process of writing this book. I am very grateful to Danuta Marek, Ewa Nawrocka, Joanna Radke, and Jerzy Nawrocki for providing access to publications not so easily accessible in my workplace. I direct my special thanks to Krystyna Ciesielska, and Jan Kniat for their efforts in improving the quality of this book.

8 viii Preface I am also very indebted to Jacek Błażewicz, Lixin Gao, Klaus Jansen, Arnaud Legrand, Cynthia Phillips, Christope Picouleau, Thomas Robertazzi, Arnold Rosenberg, Frédéric Vivien, Yakov Zinder, for providing and discussing some of their old and more recent publications. Writing this book was partially supported by grant NN of Polish Ministry of Science and Higher Education. October 2008 Maciej Drozdowski

9 Contents Preface... v 1 Introduction Field of Scheduling for Parallel Processing Basic Scheduling Notions Scheduling Theory Notions Computer Systems Notions Why We Need Scheduling Problems, Models, Algorithms, and Schedules... 5 References Basics Selected Definitions Form Graph Theory Methodology of Complexity Theory Problems, Machines, Complexity Functions Basic Complexity Classes Approximability of Hard Problems Solving Hard Combinatorial Problems Branch and Bound Algorithm Dynamic Programming Linear Programming Metaheuristics Parallel Performance Metrics References Vision of Scheduling in Parallel Systems Hardware CPU Memory Interconnection Programming Environments ix

10 x Contents 3.3 Runtime Environments Operating System Peculiarities Resource Managers References Classic Scheduling Theory Definitions jˇj Notation and Complexity Inference Scheduling Parallel Processors C max Criterion L P max Criterion cj ; P w j c j Criteria Beyond the Classics New Criteria Multicriteria Scheduling Online Scheduling Remarks on the Classic Scheduling Theory References Parallel Tasks Parallel Tasks in Practice Parallel Applications Bandwidth and Storage Allocation Reliable Computing Assumptions and Definitions Types of Parallel Tasks Processing Time Functions Extension of jˇj Notation Rigid Tasks C max ;L max Criteria Minsum Criteria Other Heuristics for Rigid Task Scheduling Moldable Tasks C max Criterion Minsum Criteria Other Heuristics for Moldable Task Scheduling Malleable Tasks C max ;L max ; max w j s j Criteria Minsum Criteria Other Heuristics for Malleable Task Scheduling Tasks with Hypercube Shape Subcube Allocation C max ;L max Criteria Minsum Criteria Other Heuristics for Tasks with Hypercube Shape...153

11 Contents xi 5.7 Tasks with Mesh Shape Submesh Allocation Heuristics with a Guarantee Heuristics Without a Guarantee Multiprocessor Tasks C max ;L max Criteria Minsum Criteria Concluding Remarks on Parallel Task Model References Scheduling with Communication Delays Scheduling with Communication Delays in Practice Formulation of the Problem Notation and Preliminaries Interconnection Topology and Communication Delay Models Technical Terminology Extension of jˇj Notation Limited Processor Number and NoDuplication Hard Cases Polynomial Cases Heuristics with a Guarantee Heuristics Without a Guarantee Limited Processor Number and Duplication Complexity of the Problem Heuristics with a Guarantee Heuristics Without a Guarantee Unlimited Processor Number and No Duplication Hard Cases Polynomial Cases Heuristics with a Guarantee Heuristics Without a Guarantee Unlimited Processor Number and Duplication Hard Cases Polynomial Cases Heuristics with a Guarantee Heuristics Without a Guarantee Scheduling in Processor Networks Scheduling in LogP Model Notation Hard Cases Polynomial Cases Heuristics with a Guarantee Heuristics Without a Guarantee Remarks on Scheduling in LogP Model...285

12 xii Contents 6.10 Scheduling with Hierarchical Communication Problem Formulation and Notation Complexity of the Problem Heuristics with a Guarantee Further Reading and Conclusions Other Branches of Scheduling with Communication Delays Observations on Scheduling with Communication Delays References Divisible Loads Star Basic Formulation Base Model Start-Up Times, Message Sequencing, Complexity Communication Options Equivalent Speed Interconnection Topologies Chains Trees Meshes Hypercubes Arbitrary Graphs Multi-installment Processing Memory Constraints Single Installment Processing Multi-installment Processing Processing Cost Optimization Negligible Start-Up Times Nonzero Start-Up Times Multiple Tasks Single Source Loads Multiple Source Load Time-Varying Environment Expected Search Time Steady-State Divisible Load Scheduling Single Originator Multiple Originators Online Scheduling Measuring Heuristics Loop Scheduling Other Heuristics...343

13 Contents xiii 7.11 Toward Discrete Load Granularity Rounding Techniques Discretely Divisible Loads DLT and Performance Evaluation DLT and the Classic Performance Measures Equivalent Processors and Ultimate Performance Isoefficiency Divisible Loads in Practice Divisible Applications Empirical Confirmations of DLT Concluding Remarks on DLT References Back to Scheduling Models On Scheduling Models What Is a Parallel Application? Parallel Systems in Scheduling Models On Bridging the Models The Case of Granularity Criteria and Constraints On Scheduling Algorithms Computational Costs Where Is My Scheduling Information? It Is a Matter of Time Toward Scheduling Problem Taxonomy Anyway? References Appendix A Summary of the Notation Index...381