PERFORMANCE EVALUATION, PREDICTION AND VISUALIZATION OF PARALLEL SYSTEMS

Transcription

1 PERFORMANCE EVALUATION, PREDICTION AND VISUALIZATION OF PARALLEL SYSTEMS

2 The Kluwer International Series on ASIAN STUDIES IN COMPUTER AND INFORMATION SCIENCE Series Editor Kai-Yuan Cai Beijing University of Aeronautics Beijing, CHINA Editorial Advisory Board Han-Fu Chen, Institute of System Science, Chinese Academy of Sciences Jun-Liang Chen, Beijing University of Post and Telecommunication Lin Huang, Peking University Wei Li, Beijing University of Aeronautics and Astronautics Hoi-Min Lin, Institute of Software Technology, Chinese Academy of Sciences Zhi-Yong Liu, Institute of Computing Technology, Chinese Academy of Sciences Ru-Qian Lu, Institute of Mathematics, Chinese Academy of Sciences Shi-Tuan Shen, Beijing University of Aeronautics and Astronautics Qing-Yun Shi, Peking University You-Xian Sun, Zhejiang University Lian-Hua Xiao, National Natural Science Foundation of China Xiao-Hu You, Southeast University Bo Zhang, Tsinghua University Da-Zhong Zbeng, Tsinghua University Bing-Kun Zhou, Tsinghua University Xing-Ming Zhou, Changsha University of Technology Also in the Series: SOFTWARE DEFECT AND OPERATIONAL PROFILE MODELING by Kai-Yuan Cai; ISBN: FUZZY LOGIC IN DATA MODELING: Semantics, Constraints, and Database Design by Guoqing Chen; ISBN: ROBUST MODEL-BASED FAULT DIAGNOSIS FOR DYNAMIC SYSTEMS by lie Chen and Ron l. Patton; ISBN:

3 PERFORMANCE EV ALUATION, PREDICTION AND VISUALIZATION OF PARALLEL SYSTEMS by Xingfu Wu Department of Computer Science, Louisiana State University, USA State Key Laboratory for Novel Software Technology at Nanjing University, China SPRINGER SCIENCE+BUSINESS MEDIA, LLC

4 ISBN ISBN (ebook) DOI / Library of Congress Cataloging-in-Publication Data A C.I.P. Catalogue record for this book is available from the Library of Congress. Copyright 1999 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in, New York 1999 Softcover reprint of the hardcover Ist edition 1999 AU rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC. Printed on acid-free paper.

5 Dedicated to my parents, my wife, and my son Zhouyang Wu.

6 SERIES EDITOR'S ACKNOWLEDGMENTS I am pleased to acknowledge the assistance to the editorial work by Beijing University of Aeronautics and Astronautics and the National Natural Science Foundation of China. Kai- Yuan Cai Series Editor Department of Automatic Control Beijing University of Aeronautics and Astronautics Beijing China

7 CONTENTS LIST OF FIGURES LIST OF TABLES PREFACE ACKNOWLEDGMENTS xi xiii xv xxi 1 INTRODUCTION Overview The Need for Grand Challenge Applications An Overview of Hardwares for Parallel Systems An Overview of Softwares for Parallel Systems Performance Issues for Parallel Systems The Organization of the Book 30 REFERENCES 32 2 SPEEDUP Overview Runtime Parallel Overhead Some Basic Parameters Speedup 44

8 Vlll Contents 2.6 Generalized Speedup 2.7 Efficiency REFERENCES S 6 SCALABILITY 6S 3.1 Overview Definitions of Scalability Metrics and Characteristics of Scalability Classification and Expressions of Scalability Scalability Analysis of Parallel Matrix Multiplication Algorithms Relations Between Scalability and Speedup 98 REFERENCES 99 PARALLEL PERFORMANCE MEASUREMENT AND ANALYSIS Overview Sequential Performance Measurement Characterizing Parallel Programs Methodology of Parallel Performance Measurement Benchmark Oriented Parallel Performance Test and Analysis Measuring Speedup and Scalability 159 REFERENCES 161 PARALLEL PERFORMANCE PREDICTION Overview Difficulties in Parallel Performance Prediction Architecture-Oriented Performance Prediction Application-Oriented Performance Prediction Predicting Speedup and Scalability 186 REFERENCES 189 PARALLEL PERFORMANCE VISUALIZATION Overview Performance Data Collection Performance Data Filtering Performance Data Visualization 218

9 CONTENTS IX 6.5 Development and Classification of Performance Visualization Tools REFERENCES PARALLEL PERFORMANCE DEBUGGING Overview Parallel Compiling-Based Performance Debugging Search-Based Performance Debugging Knowledge-Based Performance Debugging Visual Programming-Based Performance Debugging 263 REFERENCES 278 PERFORMANCE EVALUATION OF INTERCONNECTION NETWORKS FOR PARALLEL SYSTEMS 8.1 Overview An Overview of Interconnection Networks for Parallel Systems Interconnection Network Architectures Performance Metrics and Modeling for Interconnection Networks Scalability Analysis of Interconnection Networks REFERENCES INDEX 313

10 LIST OF FIGURES 1.1 Parallel generation of fractal image for Mandelbrot set A general model of a shared memory multiprocessor system A general model of a distributed memory multicomputer system A general model of a massively parallel processor system A typical architecture of a cluster of multiple computers Amdahl's Law Time-fixed speedup model Scalability metrics of parallel algorithm-architecture combinations Speedups of various scalability models Levels of parallelism Six main steps leading to loss of parallelism The SCALA framework of the SCS Group The components of a performance visualization tool The stages of software instrumentation The ASCII representation of a sample SDDP file A collection of ParaGraph utilization displays 220

11 Xli List of Figures 6.5 A collection of ParaGraph communication displays A collection of ParaGraph task displays and other displays Network View and Message Queue View in SPCview The inter-processor communication displays of VT The Source Code display of VT The display of pmview The principal structure components of a parallelizing compiler An example of HeNCE programs HeNCE node types The user graphical interface of the HeNCE CODE node types The user graphical interface of the CODE The user graphical interface of the VPE Computation graph of an example program The process of a message passing program Computation graph of the message passing Poor performance of two examples Various static interconnection topologies A shared-bus multiprocessor system A crossbar interconnection network A 8x8omega network topology A LAN simulated model 301

12 LIST OF TABLES 1.1 The comparison of MPPs and workstations with the same or comparable microprocessor Comparison of three shared memory programming standards Effect of optimizers and precompilers on naive 512 x 512 matrix-matrix multiply The speed, average speed, and utilization of three ST AP program on a 256-node SP Scalability of various architectures for the Parity Calculation The rank of four parallel computers by peak Gflops ratings The rank of four parallel computers by sustained Gflops ratings Problem sizes of NAS Parallel Benchmarks Performance results of Class A NAS Parallel Benchmarks Performance results of Class B NAS Parallel Benchmarks List of tools across various platforms and programming environments Visual parallel programming environments which support performance analysis 241

13 xiv List of Tables 6.3 Parallel programming environments which support performance analysis Classification of interconnected processors by their physical sizes 284

14 PREFACE Parallel programming differs from sequential programming in that parallel programmers must understand the large scale structure of their parallel programs in order to understand the execution performance of these parallel programs. This is a vital issue since high performance is the major reason for the existence of parallel computing, and an inefficient but otherwise correct parallel program is of practically no use for execution on a large scale parallel system. Performance evaluation, prediction and visualization of parallel systems have been the subject of many Ph.D theses, technical articles, journal special issues, and international conferences and workshops. They have also been an important component of several environments that have been developed for parallel programming, debugging, and monitoring, as well as integrated environments that combine several of these components. There are many books on parallel computing, but few detailed discussions about performance evaluation, prediction, and visualization are included in them. Three books (edited volumes) consisting of many technical papers in performance evaluation and visualization of parallel systems have been published: Multiprocessor Performance Measurement and Evaluation (Eds.: L.N. Bhuyan and Xiaodong Zhang, IEEE Computer Society Press, 1995), Performance Measurement and Visualization of Parallel Systems (Eds.: G. Haring and G. Kotsis, Advances in Parallel Computing, Vo1.7, North-Holland, 1993), and Performance Evaluation of Supercomputers (Ed.: J.L. Martin, North Holland, 1988). To the author's best knowledge, however, no book has been exclusively devoted to performance evaluation, prediction and visualization of parallel systems. This book, Performance Evaluation, Prediction and

15 xvi Preface Visualization of Parallel Systems, provides a comprehensive and systematic discussion of principles, methods, techniques and tools in performance evaluation, prediction and visualization of parallel systems. In general, manufacturers of parallel computers unilaterally follow after the peak MIPS and MFLOPS of parallel and distributed systems, and use them to evaluate the performance of parallel and distributed systems. The problem is that only a fraction of the peak performance is achievable in real benchmarks. It is estimated that many modem supercomputers and parallel computers deliver only to percent or less of their peak-performance potential in a large variety of applications. Therefore, observing the sustained performance makes more sense in evaluating parallel computer performance. Two types of metrics are widely used to measure the performance of parallel systems: parallel speedup and scalability. This book makes a thorough study of these two metrics of parallel systems. In other words, one disappointing contrast in parallel systems happens between the peak performance of the system and the actual performance of parallel applications. As parallel systems scale in size and complexity, this contrast becomes more and more important, which justifies the search for methods, tools and techniques that allow the programmers to understand the sources of performance degradation. Understanding performance is important not only for improving the efficiency of applications, but also for guiding enhancements to the architecture and the programming environment. As the number and computational power of processors in parallel computers increases, the volume and complexity of performance data gathered from these machines explodes. First, this wealth of information is a problem for the programmer who is forced to navigate through it, and for the tools that must store and process it. What makes this situation worse is that most of the time, most of the data are irrelevant to understanding the performance of the application. A second problem with the volume of the performance data available is the cost of collecting it. To minimize perturbation, a performance tool should collect only the data necessary to explain the performance of the program. However, deciding which data are relevant during a program's execution is difficult. Information overload is a serious problem facing parallel programmers. Existing performance tools incorporate several different techniques (e.g. visualization, auralization, and metrics) to address this problem. Visualization and auralization provide a way for programmers to perceive the way their program is performing via graphics and sound. Metrics use measurement to quantify the location of a bottleneck by computing numeric values for different

16 PREFACE xvii parts (e.g. procedures, processes) of the program being measured. By using these techniques, it is possible to provide programmers with insight into the performance of their program with a relatively small amount of data. To address the information overload problem, the book provides decision support that helps users to find the right data to understand their program's performance and to select appropriate ways to display and analyze it. One goal of the book is to give programmers specific advice about where the performance bottlenecks lie in a parallel program. Since this book is better described as an attempt to use both performance measurement and analytic modeling to provide performance prediction tools for parallel programmers, it characterizes related work in terms of two areas: Performance measurement techniques and tools for parallel programs typically provide a description of an execution in a form that is likely to explain reasons for poor performance and intend to suggest ways to improve the applications. They provide large amounts of information for the user so that the user can understand the application well enough to form informal mental models of the applications. Performance analysis techniques for parallel programs study the sensitivity of application performance as a function of the system size and problem size. Performance prediction is important in achieving efficient execution of parallel programs. In order to be useful to parallel programmers, performance prediction should result in predictions of actual runtime, be as automatic as possible, and be applicable to most or all parallel programs and machines. Performance prediction techniques vary depending on the type of input data used. Static techniques are the foundation of prediction, they use source code as their main input; the programmer considers a program structure and desires to know what its performance might be. Dynamic techniques rely mainly upon performance measurements, and use them to predict other performance metrics. They capture the costs specific to the particular run-time environment and machine used. Any ability to predict the performance of an application in advance of execution requires a model of the application's performance. Models can be developed in two ways: they can result from an analysis of the algorithm or program - static analysis, or from inference based on known values of the application's performance function - dynamic analysis. Although correctness debugging is an essential part of the development process for parallel programs, it is not addressed in this book. The principles required for efficient debugging of parallel programs have been better understood than those for efficient performance debugging of parallel programs. The reason seems to be that correct execution is a necessary precondition of

17 xviii Preface efficient execution, and hence initial research efforts naturally focused more heavily on debugging. In addition, research in debugging for parallel programs has developed techniques such as deterministic replay that enable programmers to debug parallel programs using methods similar to those used on sequenital programs. In contrast, techniques for performance debugging of parallel programs have not been able to make significant use of methods developed for uniprocessing. As a result of these reasons, the book in depth discusses parallel compiling-based, search-based and knowledge-based performance debugging, which assist programmers to optimize the strategy or algorithm in their parallel programs, and presents the visual programming-based performance debugging. When hundreds or thousands of computers are connected together, complex interactions, with unforeseen consequences, are common. This complexity often leads to poor performance and no one knows exactly why. The communication efficiency of interconnection network plays a central role in the performance of a parallel system. The book examines many issues related to network performance to see what kinds of problems exist and what can be done about them. It also analyzes the scalability of interconnection networks for parallel systems, and describes how to measure and improve network performance. The book is organized as follows. Chapter I gives a brief overview of performance degradation of parallel systems, and presents a general discussion of performance evaluation, prediction and visualization of parallel systems and their importance. Chapter 2 starts with analyzing and accurately defining several kinds of serial and parallel runtime, addresses some of the weaknesses of parallel speedup metrics, and discusses how to overcome them. Chapter 3 describes formal definitions of scalability, addresses the basic metrics affecting the scalability of parallel systems, discusses scalability of parallel systems from three aspects: scalability of parallel architecture, parallel algorithm and parallel algorithm-architecture combinations, and analyzes the relations of scalability and speedup. Chapter 4 discusses the methodology of performance measurement, describes the benchmark-oriented performance test and analysis and how to measure speedup and scalability in practice. Chapter 5 analyzes the difficulties in performance prediction, discusses the application-oriented and architectureoriented performance prediction and how to predict speedup and scalability in practice. Chapter 6 discusses performance visualization techniques and tools for parallel systems from three stages: techniques of performance data collection, performance data filtering and performance data visualization, and classifies the existing performance visualization tools described in this book. Chapter 7 indepth describes parallel compiling-based, search-based and knowledge-based performance debugging, which assist programmers to optimize the strategy or

18 PREFACE xix algorithm in their parallel programs, and presents the visual programming-based performance debugging. Performance debugging helps users identify the location and cause of the performance problem, and provides concrete suggestions on how to modify their parallel program in order to improve the performance. Chapter 8 presents an overview of current interconnection networks for parallel systems, analyzes the scalability of interconnection networks, discusses the ability to a network to be modularly expandable with a scalable performance with increasing machine resources, and describes how to measure and improve the network performance. For all technical contacts, suggestions, corrections, or exchanges of information, the reader is advised to contact the author via wu-xing{u@usa.net, wuxf@drum.ncic.ac.cn, or wuxf@bit.csc.lsu.edu. Xingfu Wu

19 ACKNOWLEDGMENTS This book is a revised and extended version of my Ph.D thesis. Much of the extended material in this book includes the research works of many researchers and developers allover the world, and lowe a debt of gratitude to them. As the subject area evolves rapidly, omissions are almost unavoidable. I apologize to those whose valuable work has not been included in the book. Many thanks are due to Prof. Wei Li who supervised my Ph.D thesis. After joining the National Research Center for Intelligent Computing Systems and National Research Center for High Performance Computers, Chinese Academy of Sciences, I have had more contacts and communion with parallel system researchers and developers around the world, and have learned more during this period than ever before. I thank Professors Mingfa Zhu, Guojie Li, and Zhiwei Xu of the National Research Center for Intelligent Computing Systems and National Research Center for High Performance Computers, Prof. L.M. Patnaik of India Institute of Science, Prof. Ming Pei of Michigan State University, and Prof. Kai Hwang of the University of Hong Kong for their suggestions or reviews. Prof. Xian-He Sun of Louisiana State University invited me to work with his scalable computing software group as a visiting professor for the academic year. Thanks a lot for his kind help. I appreciate the kind help from Kluwer Academic Publishers (Asian Studies in Computer and Information Science) series editor Prof. Kai-Yuan Cai, and Kluwer people Scott E. Delman and Sharon Fletcher. I am indebted to my wife Hui Yang for her constant encouragement, assistance and care during the course of writing the book. This work is supported partly from the National Research Center for Intelligent Computing Systems and National Research Center for High

20 xxii Acknowledgments Performance Computers, Chinese Academy of Sciences, the scalable computing software research group of Computer Science Department, Louisiana State University, and the State Key Laboratory for Novel Software Technology at Nanjing University, China.