Parallel Computing. Parallel Computing. Hwansoo Han

Transcription

1 Parallel Computing Parallel Computing Hwansoo Han

2 What is Parallel Computing? Software with multiple threads Parallel vs. concurrent Parallel computing executes multiple threads at the same time on multiple processors performance Concurrent computing can either alternate multiple threads on a single processor or execute in parallel on multiple processors convenient design 2

3 What is Parallel Architecture? Machines with multiple processors Intel Quad Core i7 Sony Playstation 4 IBM Blue Gene Watson Parallel Computing vs. Distributed Computing Google Data Center Locations Facebook Data Center 3

4 What is Parallel Architecture? One definition A parallel computer is a collection of processing elements that cooperate to solve large problems fast Some general issues Resource Allocation Data access, Communication and Synchronization Performance and Scalability 4

5 Why Study Parallel Computing? 10 years ago Some important applications demand high performance With CPU clock frequency scaling, it is not enough Parallel computing provides higher performance Today Many interesting applications demand high performance CPU clock rates are no longer increasing! Instruction-level parallelism is not increasing either! Parallel computing is the only way to achieve higher performance in the foreseeable future 5

6 Why Study Parallel Architecture? Role of a computer architect To design and engineer the various levels of a computer system to maximize performance and programmability within limits of technology and cost Parallelism Provides alternative to faster clock for performance Applies at all levels of system design 6

7 Why Parallel Computing? Application demands Architectural Trends The impact of technology and power 7

8 Application Trends A positive feedback cycle between the two Delivered performance Applications demand for performance New Applications More Performance Example application domains Scientific computing: CFD, biology, meteorology, General purpose computing: video, graphics, Commercial computing: databases, 8

9 Speedup Goal of applications in using parallel machines Speedup (p processors) = Performance (p processors) Performance (1 processor) For a fixed problem size (input data set), performance = 1/time Speedup fixed problem (p processors) = Time (1 processor) Time (p processors) 9

10 Scientific Computing Demand 10

11 High Performance Computing (HPC) Terascale computing (10 12 ) IBM Watson Won in Jeopardy (2011) Hadoop, Distributed computing Petascale computing (10 15 ) Tianhe-2 (Top500 #1 since 2013) Difficulties in SW development (3.5GHz POWER7 x 2,880 threads) "It's like a giant with a super body but without the software to support its thinking soul," Chi said. Some users would need years or even a decade to write the necessary code, he added. [South China Morning Post June 30, 2014] Exascale computing (10 18 ) 2020? (16,000 nodes, 2 Xeon CPU s + 3 Xeon Phi s ) 11

12 Engineering Computing Large parallel machines are mainstays in many industries Petroleum reservoir analysis Automotive crash simulation, drag analysis, combustion efficiency Aeronautics airflow analysis, engine efficiency, structural mechanics, electromagnetism Computer-aided design VLSI layout Pharmaceuticals molecular modeling Visualization all the above, entertainment, architecture Financial modeling yield and derivative analysis 12

13 Commercial Computing Relies on parallelism for high end Computational power determines scale of business that can be handled e.g.) databases, online-transaction processing, decision support, data mining, data warehousing... TPC benchmarks TPC-C (order entry), TPC-D (decision support) Explicit scaling criteria provided Size of enterprise scales with size of system Problem size no longer fixed as p increases, so throughput is used as a performance measure, transactions per minute (tpm) 13

14 Commercial Computing Hardware threads in the Top-10 TPC-C machines 14

15 Why Parallel Computing? Application demands Architectural Trends The impact of technology and power 15

16 Architectural Trends Architecture translates technology s gifts into performance and capability Four generations of architectural history Tube, transistor, IC, VLSI Greatest delineation in VLSI has been in type of parallelism exploited 16

17 Architectural Trends Greatest trend in VLSI generation is Increase in parallelism Up to 1985: bit level parallelism 4-bit 8 bit 16-bit, slows after 32 bit Adoption of 64-bit now under way, 128-bit far (not performance issue) Great inflection point when 32-bit micro and cache fit on a chip Mid 80s to mid 90s: instruction level parallelism Pipelining and simple instruction sets + compiler advances (RISC) On-chip caches and functional units superscalar execution Greater sophistication: out of order execution, speculation, prediction To deal with control transfer and latency problems Since 2002: thread level parallelism Multicore processors 17

18 Transistors Phases in VLSI Generation 100,000,000 Bit-level parallelism Instruction-level Thread-level (?) 10,000,000 1,000, ,000 4bits 8bits 16bits 32bits i80286 R10000 Pentium i80386 R2000 R3000 Multicore / manycore Heterogenous multicore GPGPU 10,000 i8086 i8080 i8008 i4004 Pipelining Superscalar Branch prediction Out-of-order execution VLIW 1,

19 Fraction of total cycles (%) Speedup Limitation in ILP Number of instructions issued Instructions issued per cycle(issue width) Speedup begins to saturate after issue width of 4 Assumption of ideal machine Infinite resources and fetch bandwidth, perfect branch prediction and renaming But with realistic memory Real caches and non-zero miss latencies 19

20 Single-Thread Performance Curve Architecture -driven Technology -driven The rate of single-thread performance improvement has decreased [Computer Architectures Hennessy & Patterson] 20

21 Why Parallel Computing? Application demands Architectural Trends The impact of technology and power 21

22 Desperate Cooling? 22

23 Cooling Power delivery, packaging, and cooling costs Increased cost of thermal packing $1 per watt for CPUs more than 35W [Tiwari, DAC98] At high-end, 1W of cooling for every 1W of power Cooling Physical solutions Heatsink, thermal paste, fan APM (advanced power management) BIOS manages clock speed Triggered by thermal diodes ACPI (Advanced Configuration and Power Interface) OS manages power for all devices Idle, nap, sleep modes 23

24 Power Consumption Trend 1000 Watts/cm Nuclear Reactor Rocket Nozzle 10 Hot plate 1 Reason for power trends Smaller feature size Increased compaction High clock frequency More complicated processors Process Transistors Supply voltage Frequency Power nm 90nm 55nm 3nm 0.2B 1.2B 7.8B 50B V V 0.7V 0.5V 1GHz 3GHz 10GHz 30GHz 100W 175W 300W 540W source: Intel (2002) 24

25 Power Consumption (watts) Core2Quad 3.0GHz Core2duo 3.0GHz Core2duo 2.4GHz (mobile) Core2duo 1.2GHz (mobile) Source: MIT IAP 25

26 Recent Intel Processors The future is not higher clock rate, but multiple cores per die We are dedicating all of our future product development to multicore designs. We believe this is a key inflection point for the industry. - Paul Otellini, Intel President (IDF 2005) year Transistors Clock(GHz) Power(W) Pentium M 1.3~3.8 21~115 Pentium M M 0.8~ ~27 Core Duo M 1.06~ ~49 Core 2 Duo M 1.06~ ~65 Core 2 Quad M 2.1~ ~150 Core i7 (Quad) M 1.6~3.5 45~130 Xeon ~ ~130 source: wikipedia : comparison of Intel processors 26

27 Summary: Why Parallel Computing? If you can t exploit parallelism, performance will be a zero sum game If you want to add a new feature and maintain performance, you will need to remove something else. The future is multicore (i.e. parallel) according to all major processor vendors We expect more and more cores will be delivered with each generation Starting now, everyone will need to know how to write parallel programs It is no longer a niche area: it is a mainstream. 27