Algorithms and Parallel Computing

Size: px

Start display at page:

Download "Algorithms and Parallel Computing"

Hugo Greer
6 years ago
Views:

3 Algorithms and Parallel Computing

5 Algorithms and Parallel Computing Fayez Gebali University of Victoria, Victoria, BC A John Wiley & Sons, Inc., Publication

6 Copyright 2011 by John Wiley & Sons, Inc. All rights reserved Published by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) , fax (978) , or on the web at Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) , fax (201) , or online at permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) , outside the United States at (317) or fax (317) Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at Library of Congress Cataloging-in-Publication Data Gebali, Fayez. Algorithms and parallel computing/fayez Gebali. p. cm. (Wiley series on parallel and distributed computing ; 82) Includes bibliographical references and index. ISBN (hardback) 1. Parallel processing (Electronic computers) 2. Computer algorithms. I. Title. QA76.58.G dc Printed in the United States of America

7 To my children: Michael Monir, Tarek Joseph, Aleya Lee, and Manel Alia

9 Contents Preface xiii List of Acronyms xix 1 Introduction Introduction Toward Automating Parallel Programming Algorithms Parallel Computing Design Considerations Parallel Algorithms and Parallel Architectures Relating Parallel Algorithm and Parallel Architecture Implementation of Algorithms: A Two-Sided Problem Measuring Benefits of Parallel Computing Amdahl s Law for Multiprocessor Systems Gustafson Barsis s Law Applications of Parallel Computing 22 2 Enhancing Uniprocessor Performance Introduction Increasing Processor Clock Frequency Parallelizing ALU Structure Using Memory Hierarchy Pipelining Very Long Instruction Word (VLIW) Processors Instruction-Level Parallelism (ILP) and Superscalar Processors Multithreaded Processor 49 3 Parallel Computers Introduction Parallel Computing Shared-Memory Multiprocessors (Uniform Memory Access [UMA]) Distributed-Memory Multiprocessor (Nonuniform Memory Access [NUMA]) 56 vii

10 viii Contents 3.5 SIMD Processors Systolic Processors Cluster Computing Grid (Cloud) Computing Multicore Systems SM Communication Between Parallel Processors Summary of Parallel Architectures 67 4 Shared-Memory Multiprocessors Introduction Cache Coherence and Memory Consistency Synchronization and Mutual Exclusion 76 5 Interconnection Networks Introduction Classification of Interconnection Networks by Logical Topologies Interconnection Network Switch Architecture 91 6 Concurrency Platforms Introduction Concurrency Platforms Cilk OpenMP Compute Unified Device Architecture (CUDA) Ad Hoc Techniques for Parallel Algorithms Introduction Defining Algorithm Variables Independent Loop Scheduling Dependent Loops Loop Spreading for Simple Dependent Loops Loop Unrolling Problem Partitioning Divide-and-Conquer (Recursive Partitioning) Strategies Pipelining Nonserial Parallel Algorithms Introduction Comparing DAG and DCG Algorithms Parallelizing NSPA Algorithms Represented by a DAG 145

11 Contents ix 8.4 Formal Technique for Analyzing NSPAs Detecting Cycles in the Algorithm Extracting Serial and Parallel Algorithm Performance Parameters Useful Theorems Performance of Serial and Parallel Algorithms on Parallel Computers z-transform Analysis Introduction Definition of z-transform The 1-D FIR Digital Filter Algorithm Software and Hardware Implementations of the z-transform Design 1: Using Horner s Rule for Broadcast Input and Pipelined Output Design 2: Pipelined Input and Broadcast Output Design 3: Pipelined Input and Output Dependence Graph Analysis Introduction The 1-D FIR Digital Filter Algorithm The Dependence Graph of an Algorithm Deriving the Dependence Graph for an Algorithm The Scheduling Function for the 1-D FIR Filter Node Projection Operation Nonlinear Projection Operation Software and Hardware Implementations of the DAG Technique Computational Geometry Analysis Introduction Matrix Multiplication Algorithm The 3-D Dependence Graph and Computation Domain D The Facets and Vertices of D The Dependence Matrices of the Algorithm Variables Nullspace of Dependence Matrix: The Broadcast Subdomain B Design Space Exploration: Choice of Broadcasting versus Pipelining Variables Data Scheduling Projection Operation Using the Linear Projection Operator Effect of Projection Operation on Data The Resulting Multithreaded/Multiprocessor Architecture Summary of Work Done in this Chapter 207

12 x Contents 12 Case Study: One-Dimensional IIR Digital Filters Introduction The 1-D IIR Digital Filter Algorithm The IIR Filter Dependence Graph z-domain Analysis of 1-D IIR Digital Filter Algorithm Case Study: Two- and Three-Dimensional Digital Filters Introduction Line and Frame Wraparound Problems D Recursive Filters D Digital Filters Case Study: Multirate Decimators and Interpolators Introduction Decimator Structures Decimator Dependence Graph Decimator Scheduling Decimator DAG for s 1 = [1 0] Decimator DAG for s 2 = [1 1] Decimator DAG for s 3 = [1 1] Polyphase Decimator Implementations Interpolator Structures Interpolator Dependence Graph Interpolator Scheduling Interpolator DAG for s 1 = [1 0] Interpolator DAG for s 2 = [1 1] Interpolator DAG for s 3 = [1 1] Polyphase Interpolator Implementations Case Study: Pattern Matching Introduction Expressing the Algorithm as a Regular Iterative Algorithm (RIA) Obtaining the Algorithm Dependence Graph Data Scheduling DAG Node Projection DESIGN 1: Design Space Exploration When s = [1 1] t DESIGN 2: Design Space Exploration When s = [1 1] t DESIGN 3: Design Space Exploration When s = [1 0] t Case Study: Motion Estimation for Video Compression Introduction FBMAs 256

13 Contents xi 16.3 Data Buffering Requirements Formulation of the FBMA Hierarchical Formulation of Motion Estimation Hardware Design of the Hierarchy Blocks Case Study: Multiplication over GF(2 m ) Introduction The Multiplication Algorithm in GF(2 m ) Expressing Field Multiplication as an RIA Field Multiplication Dependence Graph Data Scheduling DAG Node Projection Design 1: Using d 1 = [1 0] t Design 2: Using d 2 = [1 1] t Design 3: Using d 3 = [1 1] t Applications of Finite Field Multipliers Case Study: Polynomial Division over GF(2) Introduction The Polynomial Division Algorithm The LFSR Dependence Graph Data Scheduling DAG Node Projection Design 1: Design Space Exploration When s 1 = [1 1] Design 2: Design Space Exploration When s 2 = [1 0] Design 3: Design Space Exploration When s 3 = [1 0.5] Comparing the Three Designs The Fast Fourier Transform Introduction Decimation-in-Time FFT Pipeline Radix-2 Decimation-in-Time FFT Processor Decimation-in-Frequency FFT Pipeline Radix-2 Decimation-in-Frequency FFT Processor Solving Systems of Linear Equations Introduction Special Matrix Structures Forward Substitution (Direct Technique) Back Substitution Matrix Triangularization Algorithm Successive over Relaxation (SOR) (Iterative Technique) Problems 321

14 xii Contents 21 Solving Partial Differential Equations Using Finite Difference Method Introduction FDM for 1-D Systems 324 References 331 Index 337

15 Preface ABOUT THIS BOOK There is a software gap between hardware potential and the performance that can be attained using today s software parallel program development tools. The tools need manual intervention by the programmer to parallelize the code. This book is intended to give the programmer the techniques necessary to explore parallelism in algorithms, serial as well as iterative. Parallel computing is now moving from the realm of specialized expensive systems available to few select groups to cover almost every computing system in use today. We can find parallel computers in our laptops, desktops, and embedded in our smart phones. The applications and algorithms targeted to parallel computers were traditionally confined to weather prediction, wind tunnel simulations, computational biology, and signal processing. Nowadays, just about any application that runs on a computer will encounter the parallel processors now available in almost every system. Parallel algorithms could now be designed to run on special - purpose parallel processors or could run on general - purpose parallel processors using several multilevel techniques such as parallel program development, parallelizing compilers, multithreaded operating systems, and superscalar processors. This book covers the first option: design of special - purpose parallel processor architectures to implement a given class of algorithms. We call such systems accelerator cores. This book forms the basis for a course on design and analysis of parallel algorithms. The course would cover Chapters 1 4 then would select several of the case study chapters that constitute the remainder of the book. Although very large - scale integration (VLSI) technology allows us to integrate more processors on the same chip, parallel programming is not advancing to match these technological advances. An obvious application of parallel hardware is to design special - purpose parallel processors primarily intended for use as accelerator cores in multicore systems. This is motivated by two practicalities: the prevalence of multicore systems in current computing platforms and the abundance of simple parallel algorithms that are needed in many systems, such as in data encryption/ decryption, graphics processing, digital signal processing and filtering, and many more. It is simpler to start by stating what this book is not about. This book does not attempt to give a detailed coverage of computer architecture, parallel computers, or algorithms in general. Each of these three topics deserves a large textbook to attempt to provide a good cover. Further, there are the standard and excellent textbooks for each, such as Computer Organization and Design by D.A. Patterson and J.L. xiii

Real-Time Optimization by Extremum-Seeking Control

Real-Time Optimization by Extremum-Seeking Control Real-Time Optimization by Extremum-Seeking Control KARTIK B. ARIYUR MIROSLAV KRSTIĆ A JOHN WILEY & SONS, INC., PUBLICATION Copyright 2003 by John Wiley