Fundamentals of. Parallel Computing. Sanjay Razdan. Alpha Science International Ltd. Oxford, U.K.

Transcription

1 Fundamentals of Parallel Computing Sanjay Razdan Alpha Science International Ltd. Oxford, U.K.

2 CONTENTS Preface Acknowledgements vii ix 1. Introduction to Parallel Computing Parallel Computing Components of Parallel Computing System Parallel Hardware Parallel Operating System Parallel Programs Multiprocessor vs. Multi-core Architecture Why Parallelism Moore's Law Sequential vs. Parallel Computing Program Process Thread Instruction Concurrent Computing Communication between Concurrent Systems Coordinating Access to Resources Distributed Computing Scalability Redundancy Levels of Parallelism Data level Parallelism Instruction Level Parallelism Thread or Task Level Parallelism Bit Level Parallelism Considerations while Writing Parallel Programs Communication Load Balancing Synchronization Need for Parallel Programs Models of Parallel Algorithm Data Parallel Model Pipeline Model 1.30

3 xii Contents Work Pool Model Master-Slave Model Hybrid Model Types of Parallel Computing Highly Parallel Computing Massively Parallel Computing Cluster Computing Grid Computing Advantages of Parallel Computing Time and Cost Efficiency Solving Larger Problems Using Non-local Resources Application of Parallel Computing Image Processing Seismology Protein Folding Databases Search Engines Drug Discovery and Drug Design 1.36 Exercise Architecture of Parallel Computers Von Neumann Architecture Von Neumann Instructions Von Neumann Instruction Cycle Instruction and Data Stream Limitations of Von Neumann Architecture Improvements of Von Neumann Architecture Classification of Parallel Computers Flynn's Classification Parallelism at Hardware Level (Handler's Classification) Classification on the Basis of Structure Levels of Parallelism on the Basis of Grain Size Dependency and its Types Data Dependency Flow Dependency Output Dependency Anti-dependency I/O Dependency Control Dependency Resource Dependency Bernstein Conditions for Detecting Parallelism 2.21 Exercise 2.23

4 Contents xiii 3. Interconnection Topologies Purpose of Interconnection Internetworking Terminology Topology Switching Routing Flow Control Node Degree Network Diameter Bisection Width Network Redundancy Network Throughput Network Latency Hot Spot Dimension ofnetwork Broadcast and Multicast Blocking vs. Non-blocking Networks Static vs. Dynamic Network Direct vs. Indirect Interconnection Network Network Topologies Bus Topology Star Topology Linear Array Mesh Topology Ring Topology Torus Topology Fully Connected Topology Crossbar Network Topology Tree Interconnection Topology Fat Tree Topology Cube Internetwork Topology Hypercube Internetworking Shuffle Network Omega Network Butterfly Internetwork Benz Network Pyramid Network 3.25 Exercise Parallel Algorithms Algorithms Analyzing a Sequential Algorithm Big O Notation 4.3

5 XIV Contents 4.3 Analyzing Parallel Algorithms Time Complexity Cost Number of Processors Space Complexity Speedup Efficiency Scalability Amdahl's Law Cost Optimality of Parallel Algorithms Some Examples of Cost Optimal Algorithms 4.19 Exercise Graph Algorithms Graph Terminology Cyclic Graph Complete Graph Weighted Graph Shortest Path Between Vertices Data Structure to Store Graph Solving Problems with Graph Graph Traversal Prim's Algorithm Minimum Spanning Tree Single-Source Shortest Path Connected Components of a Graph 5.31 Exercise Parallel Sorting and Searching Sorting Networks Bitonic Sorting Network Merging Sorted Sequences Parallel Searching Algorithms Binary Search Algorithm Parallel Sorting Algorithms Odd-Even Swap Sort Insertion Sort Selection Sort Bubble Sort Merge Algorithm Solving Linear Equations Gaussian Elimination Method 6.21 Exercise 6.26

6 Contents XV 7. PRAM Model of Computation Model of Computation RAM Model of Computation PRAM Model of Computation Conflict Resolution Techniques PRAM Models Concurrent Read Concurrent Write (CRCW) Concurrent Read Exclusive Write (CREW) Exclusive Read Exclusive Write (EREW) Exclusive Read Concurrent Write (ERCW) PRAM Algorithms CRCW Maximum Number Algorithm CRCW Matrix Multiplication EREW Search Algorithm EREW Maximum Algorithm CREW Matrix Multiplication 7.14 Exercise Parallel Operating System Parallel Operating System Process Management Scheduling Process Synchronization Protection 8.3 Exercise Basic Data Structure Data Structure Arrays Linked List Binary Tree 9.7 Exercise Trends in Parallel Computing Parallel Operating System How PVM Works? Cluster Computing Grid Computing Grid Management Components (GMC) Donor Software Schedulers Hyper-Threading 10.6 Exercise 10.6 Index I-l