Loop Tiling for Parallelism

Transcription

1 Loop Tiling for Parallelism

2 THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE

3 LOOP TILING FOR PARALLELISM JINGLING XUE School of Computer Science and Engineering The University of New South Wales Sydney, NSW 2052, Australia ~. " Springer Science+Business Media, LLC

4 Library of Congress Cataloging-in-Publication Xue, Jingling, Loop tiling for parallelism / Jingling Xue. p. cm. -- (Kluwer international series in engineering and computer science ; SECS 575) lncludes bibliographical references and index. ISBN ISBN (ebook) DOI / Parallel processing (Electronic computers) 2. Electronic data processing--distributed processing. 3. Loop tiling (Computer science) 1. Title. II. Series. QA76.58.X '.35--dc Copyright 2000 Springer Science+Business Media New York Originally published by Kluwer Academic Publishers, N ew York in 2000 Softcover reprint ofthe hardcover Ist edition 2000 All rights reserved. No part ofthis publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission ofthe publisher, Springer Science+Business Media, LLC Printed an acid-free paper.

5 Contents List of Figures List of Tables Preface Acknowledgments ix xiii xv xix Part I Mathematic Background and Loop Transformation 1. MATHEMATICAL BACKGROUND 1.1 Logic 1.2 Sets 1.3 Arithmetic 1.4 Vectors and Matrices 1.5 Roots of Cubic and Quartic Equations 1.6 Integer Matrices 1. 7 Convex Analysis 1.8 Convex Polyhedra 1.9 Convex Cones 1.10 Fourier-Motzkin Elimination 1.11 Further Reading 2. NONSINGULAR TRANSFORMATIONS AND PERMUTABILITY Perfectly Nested Loops Dependence Vectors and Their Polyhedra Iteration-Reordering Transformations Fully Permutable Loop Nests Further Reading Part II Tiling as a Loop Transformation 3. RECTANGULAR TILING 73

6 vi Contents 3.1 Modeling Rectangular Tiling Legality Test Tile Dependences and Tile Space Graph Tiled Code Tiling-Related Transfonnations Yet Another Tiling Model Further Reading PARALLELEPIPED TILING ' Why Parallelepiped Tiling? Legality Test Tile Dependences and Tile Space Graph Tiled Code Decomposition of Parallelepiped Tiling Yet Another Tiling Model Loop Partitioning v.s. Loop Tiling Further Reading 120 Part III Tiling for Distributed-Memory Machines 5. SPMD CODE GENERATION Background Machine Model Computation Distribution Data Distribution Code Generation Message-Passing Code Generation Memory Management SPMD Code in Local Address Space SPMD Code for Parallelepiped Tiling Experiments Further Reading COMMUNICATION-MINIMAL TILING Computation & Communication Volumes Problem Fonnulation Closed-Fonn Optimal Tilings All Extremal-Ray Optimal Tilings Making H-1 Integral Further Reading TIME-MINIMAL TILING Parallelogram Tiling Executing Tiles in the SPMD Paradigm 201

7 Contents 7.3 Computation and Communication Models 7.4 Rise 7.5 Optimal Tile Size 7.6 Experiments 7.7 Further Reading Bibliography Index Vll

8 List of Figures 1.1 Plots of two cubic functions Plot of a quartic function Integer lattices Convex and affine hulls The unique minimum of a strictly convex function Convex polyhedra Sets that are not convex cones Convex cones The dual cones of the convex cones in Figure An algorithm for finding the lines and rays of a cone Fourier-Motzkin elimination and its inexactness in 7l.n Using Fourier-Motzkin elimination to scan a polytope The iteration space of Example The iteration space graphs for Example An algorithm for constructing the dependence polyhedron The dependence polyhedra for four dependence vectors Transformation of a distance vector by an injective mapping One-to-many transformation of distance vectors Transformation of a distance vector by a linear transformation The transformed iteration spaces of Example Transformations of dependence polyhedra Approximating the dependence polyhedra in Figure An algorithm for splitting a dependence vector J Finding a unimodular canonical transformation Examples of rectangular things Tile origins x 2 rectangular tiling of a double loop A non-convex tile space 78

9 x List of Figures 3.5 Tile offset v.s. loop normalisation Transformations of distance vectors Legal tiling for a non-permutable double loop Two examples tiled by the same 2 x 2 tiling Tile space graph Sequential tiled code in the 'Y model Examples of 2-D loop nests Tile codes for example loop nests in the 'Y model Strip-mining and tiling Loop skewing and rectangular tiling Two rectangular tiling models compared Sequential tiled code in the 'Y model Tile codes for example loop nests in the 'Y model A parallelogram tiling for Example Non-identical tiles under a ''n~tegral'' tiling transformation Approximation ofpp(d) by Pp(d) Sequential tiled code in the p model Decomposition of the parallelogram tiling in Figure Sequential tiled code in the p model Transformed loop nest for loop partitioning SPMD code generation A running example Sequential tiled code SPMD code after computation distribution Local data spaces for Figure The data owned by the host for Figure I/O code for receiving read-only data Communication sets for Example Two message-passing code sections for Figure Simplified message-passing code SPMD program with communication code Local memory allocation for the running example Translation of global and local loop indices Local array indices for the running example SPMD code in local address space Normalised SPMD code in local address space SPMD program for running example in local address space Iteration spaces for SOR Computation distribution for SOR Impact of tile size Computation and communication overlap Tiling, communication and validity test overheads 166

10 Contents Xl 5.23 Impact of tile shape The ISG and its 4 x 4 tiling for Example Dependence and time cones for Example Approximation of the communication volume of a tile nonlocal data accessed by a tile for Example Dependence and time cones for example Optimal tiling when D = I 2x Optimal tiling when D = (d~, d-;) E 7l. 2X Optimal tiling when D E 7l. 2xm The dependence cone for Example The dependence cone for Example A procedure for finding all extremal-ray optimal tilings Average run time of OptComTiling A geometric interpretation of Hl in Example Tiling of a parallelogram-shaped iteration space The legality of a parallelogram tiling Cyclic tile distribution over P = 4 processors Pipelining of non-constant dependences Communication cost model Rise for a tiled parallelogram iteration space Three types of rise values Derivation of execution time when rise r < Solution spaces and two separating constraints B 1 ( w) is above the right boundary of F B 1 ( w) intersects the right boundary of F A sketch of the proof for Lemma Bl (w) is below the right boundary of F A sketch of the proof for Lemma Derivation of execution time when rise r = A sketch of the proof for Lemma Derivation of execution time when rise r > Zigzag path when swp > H An algorithm for finding optimal tile size when r > Fidle. Ffree and C2 (w, h) = Communication parameters for AP x 2 rectangular tiling of 5-point SOR Performance of 5-point SOR on 10 processors Performance of 5-point SOR on 50 processors Performance results for several values of P x 2 parallelogram-shaped tiling of 3-point SOR Performance of 3-point SOR on 10 processors Performance of 3-point SOR on 50 processors 240

11 xii List of Figures 7.29 Performance of 3-point SOR on 100 processors 7.30 Performance of 3-point SOR on 61 processors 7.31 Performance of 3-point SOR on 50 processors 7.32 Plots of hi = h(wi) and hf = Fh(Wf)

12 List of Tables Shorthand notations for direction values A list of representative loop transformations 40 45

13 Preface Techniques for constructing restructuring compilers for parallel machines have been developed over the past three decades. Some of these techniques are introduced in Hans Zima's book on Supercompilers for Parallel and Vector Computers, Utpal Banerjee's book series on Loop Transformations for Restructuring Compilers, Michael Wolfe's book on High Performance Compilers for Parallel Computing, and recently, the book on Scheduling and Automatic Parallelization co-authored by Darte, Robert and Vivien. When optimising the performance of scientific and engineering programs, the most gains come from optimising nested loops or recursive procedures, where major chunks of computation are performed repeatedly. A large number of loop transformations have been accumulated over the years, and some of these can be found in research and production compilers. Loop tiling, originally promoted by Francois Irigoin and Michael Wolfe, is one of the most important iteration-reordering loop transformations. Loop tiling is beneficial for both parallel machines and uniprocessors with multilevels of cache. Together with other transformations such as loop distribution and loop fusion, loop tiling can reduce communication and synchronisation cost, maximise parallelism and improve memory hierarchy performance. Over the last few years or so, a lot of research efforts have been focussed on exploring the use of loop tiling to maximise parallelism for parallel machines or otherwise improve cache locality. Optimising for cache locality has become critically important for performance. Several research groups around the world are actively working on tackling this problem. Although progress has been made, much remains to be done. Therefore, the use of loop tiling for locality optimisations is not covered. As a consequence, some related publications are not cited in the reference list. However, the first two parts of the book provide the basic foundation useful for the general loop tiling technique. This book explores the use of loop tiling for minimising synchronisation and communication cost and maximising parallelism for parallel machines. The

14 xvi Preface book is organised into three parts. The first part, consisting of Chapters 1 and 2, provides the general mathematical background and introduces a theory of nonsingular loop transformations. Chapter 1 describes the basic mathematical concepts and tools necessary for a understanding of the subject with a particular emphasis on convex cones. Convex cones will be used throughout the book for addressing a number of important problems, including data dependence abstraction, loop permutability, legality test, and tile size and shape selection. Our treatment of nonsingular loop transformations in Chapter 2 serves to set up the context in which other iteration-reordering transformations such as loop tiling can be further developed. In particular, this chapter discusses data dependences, introduces legality test and code generation required for a nonsingular transformation, and relates the full permutability of a loop nest with the degree of parallelism and locality inherent in the loop nest. The second part, consisting of Chapters 3 and 4, deals with both rectangular and parallelepiped tiling. Tiling is discussed in terms of its effects on the data dependences and the required dependence test for legality. Unlike nonsingular loop transformations, the exact test for the legality of a tiling requires the knowledge of both the data dependences and the extent and shape of the iteration space and can be solved, in principle, by integer programming. For realistic tiling cases, efficient legality tests based on the data dependence information alone are described. This chapter also discusses the generation of tiled code and exposes the duality between loop tiling and loop partitioning. The last part, consisting of Chapters 5-7, focuses on minimising the execution time of a loop nest on a distributed memory machine. Chapter 5 describes a suite of compiler techniques for generating a SPMD program to execute a tiled iteration space. Chapter 6 addresses an interesting problem of determining the best tile shape to minimise inter-tile communication once the tile size is given. The solution to this problem provides insights for understanding various tilingrelated problems. Chapter 7 deals with the problem of finding the best tile size for a double loop once the tile shape is more or less given. The techniques presented in the last part can be adapted to work for a cluster of workstations, except that the tiles of varying sizes and a more sophisticated cost model may be needed to cope with heterogeneity present at all levels of network, processor and program. The techniques presented in the last part are also directly applicable to shared-memory machines once the machines are modeled as BSP (Bulk Synchronous Parallel) machines. In the case of the SPMD code generation, the send and receive calls can be replaced with an appropriate synchronisation mechanism. Each chapter includes a "Further Reading" section that contains citations to the original material in the reference list. JINGLING XUE

15 To my wife, Lili

16 Acknowledgments The author would like to thank all those who gave their own time and effort in the making of this book. Francois lrigoin of Ecole des Mines de Paris found time in his busy schedule to provide insightful and critical comments to my questions. The other reviewers of the book include Peizong Lee of Academia Sinica, Taiwan, Zhiyuan Li of Purdue University, Yves Robert of of Laboratoire de l'informatique du Parallelisme at Lyon and Peiyi Tang of University of Southern Queensland. I am very grateful to all these reviewers for encouraging me to write this book and for giving a number of suggestions. Alain Darte of Laboratoire de l'informatique du Parallelisme at Lyon read Chapter 1 very carefully, found errors and gave several suggestions. XIX