S16-02, URL:

Transcription

1 Self Introduction A/Prof ay Seng Chuan el: Office: S-0, Dean s s Office at Level URL: htt:// I was a rogrammer from to. I have been working in NUS since, and I teach mainly comuter and comuting subjects although h I also have taught Physics since last year. (Deuty Director at Centre for Remote Imaging Sensing and Processing: 00 to 00). I am (was) a hands-on erson on High Performance Comuting. odate I have made my hands dirty on PC-transuter System, network of workstations, and GRID System. Particiated in IBM-iHPC ihpc s s Blue Challenge in 00. Current Duties in NUS: Day: SM/SM Programme (MOE), Faculty I Unit (Science), Physics Det (Associate Professor). I am also a regular mentor of MOE Gifted Education Programme. Night: Assistant Master of emasek Hall, Resident Fellow (Block E). Consultation for CZ Day: S-0, Dean s s Office. By aointment or el at any time. It is better to give a telehone call first so that I will be waiting for you. Night: emasek Hall, Block E, Room E0. Give a call first. ock E). Continual Assessments sets of individual assignment to be graded (% x) hand written form but you can also tye it, and rograms. set of grou roject to be graded (0%) reort form (to be tyed) with rograms and resentation is needed. We ractise honour system in the award of scores, ie, you cannot coy. Exam 0% from final exam Closed book. Prerequisite Programming knowledge is a must.

2 extbooks Introduction to Parallel Comuting ( nd Edition), by Ananth Grama, Anshul Guta, George Karyis, and Viin Kumar, Addison Wesley, Second Edition, 00, ISBN Parallel Programming in C with MPI and OenMP,, by Michael J. Quinn, Mc Graw Hill, ISBN References A few scientific aers he Art of Comuter Programming, Volumes - by Donald E. Knuth, Addison-Wesley Publishing Co., October, MPI Manual oics Introduction From Hands-on Exerience Hardware Platforms Vector/Distributed S/W Platforms (threading versus message assing) MPI (Message Passing Interface) Scientific Comutation Examles (Parallel algorithms for matrix multilication, linear systems, sorting and merging) Parallel Discrete Systems CZ High Performance Comuting Lecture : Introduction From Hands-on Exerience A/Prof ay Seng Chuan Reference: exerience!!!

3 Objectives to areciate the organization of a arallel rocessing system and the communication models used in arallel comuting to understand the notions of seedu and efficiency and their imlications Objectives (cont d) to areciate the interacting effect of rocess granularity, communication overhead, load balancing and arallelization enalty to challenge common senses and common beliefs with regards to the fairness of workload distribution Let s s do some calculation Answer : Suose a comutation time of nanosecond (suercomuter range) is exected. What is the distance traveled by an electromagnetic signal on ICs? An electromagnetic signal at free sace travels at m/s (~ x m/s). he seed of a comuter is inversely roortional to the transmission delay of electrical signals on the ICs. Since distance seed x time, the distance traveled is x x - 0. m or 0 cm.

4 High Performance Comuting : he Motivation he distance calculated is reduced by a factor of to in many materials used to build comuter. As such the distance traveled is reduced to cm or less (dimension of PDA). Can we find a PDA (Personal Digital Assistant) of this size and yet is able to achieve the suercomuter erformance? Parallel Comuting : he Motivation (cont d) Answer : Not otimistic. How to resolve the cooling roblem as the ICs are so lose to each other? he marginal imrovement will be rogressively exensive. Alternative : High Performance Comuting. Parallel Comuting System A arallel comuting system is a latform that contains a collection of rocessing elements which can communicate and cooerate to solve large roblems fast. Communication Models of Parallel Comuting System. Shared Memory Shared Memory..

5 Shared Memory he shared memory configuration is not scalable as the access to the same memory location may need to be sequentialized so that data integrity can be assured. Sequentialization is equivalent to the rogram execution on a unirocessor. Shared Memory Communication Models of Parallel Comuting System (cont d). Private Memory Memory Memory Memory Each uses a segment of its rivate memory for inter-rocessor communication. Performance Metrics Seedu Seedu (S) is the ratio between the time needed for the most efficient sequential algorithm ( ) to erform a comutation, and the time needed to erform the comutation on a arallel machine incororating arallelism ( ) where rocessing elements, >,, are used. Performance Metrics (cont d) Seedu ( ) S S() means a linear seedu S() < means a sub-linear seedu S() > means a suer-linear seedu S() < means a slow down 0

6 Examle: A Search Problem In the following grid cells there is one negative number. What is the time required to find the number? - Search Problem (cont d) - Let c be the time required to rocess a cell, and a to down search on the columns is adoted. When rocessor is used, c. When rocessors are used and assume no overhead, c. Search Problem (cont d) Seedu S() c c Seculation : Can seedu be greater than (or can efficiency be greater than 0%)? Given a sequential algorithm (G) that incurs the least runtime ( ) for a roblem. Suose its arallel version is able to achieve a suer- linear seedu. We have S ( ) < >

7 Seculation (cont d) We now construct a sequential algorithm (G) by sequentializing the arallel algorithm. he runtime of G on one rocessor is < Seculation (cont d) his is an anomaly since G is the best sequential algorithm (of the least runtime) but now the runtime of G is less than. herefore, a suer-linear seedu cannot be guaranteed. But it can haen if the condition is right. terms Instance for Suer-linear Seedu Consider the grid search again. What is the seedu if the grid contents are: Comuting Communication Overhead and Comutation Granularity - c c S() c > Communication Overhead α + n x β α channel setu time n x β transmission time, where n is the number of bits, and β is time required to transmit one unit of data

8 Comuting (cont d) Consider a square grid with hea roerty father son son Comuting (cont d) Give a square grid filled with random data. he hea roerty can be established by alying row sort and column sort. he usefulness of hea roerty is that the smallest data on the square grid or sub- grids can be located at O() time. 0 After Row Sort Random Data After Column Sort Suose the time required for each row sort or column sort is s.. We have s + s s. Comuting (cont d) What if rocessors are used? Parallel Row Sort

9 Comuting (cont d) Comuting (cont d) (α( α + nβ) ) + s +(α α + nβ) + (α α + nβ) ) + s +(α α + nβ) s + (α α + nβ) Parallel Column Sort Comuting (cont d) he communication overhead should be treated as a relative term with resect to comutation granularity. Why? s S ( ) s + ( α + nβ ) If s >> α + nβ, S(). (What is $000 if you already have $,000,000!!) Otherwise S() is sub-linear, and in the worst case it can become a slowdown. Load Balancing Should workload be evenly (equally or fairly) distributed? Common belief : Yes. My answer : Not always yes if our objective is to minimize the runtime of a arallel rogram.

10 Algorithm Penalty (cont d) Algorithm enalty is of O(n) for the grid oint consolidation. High Performance Comuting : o do or not to do? High Performance Comuting introduces a new set of roblems which does not exist in the sequential version. his will be discussed in CZ. o achieve a good seedu, the interacting effect of rocess granularity, communication overhead, load balancing, and arallelization enalty must be considered. his will be discussed in CZ. here are many HPC alications that have achieved a close-to to-linear seedu. Conclusions he seed of the traditional comuter cannot be increased indefinitely. A arallel latform offers the otential to reduce rogram runtime. Parallel comuting also incurs communication overhead and arallelization enalty. An even workload distribution scheme may not result in the least runtime. Conclusions (cont d) o achieve a good seedu, the interacting effects of rocess granularity, communication overhead, load balancing, and arallelization enalty must be considered. You got to know how to do it, and I will teach you in this course. 0