Distributed systems: paradigms and models Motivations

Transcription

1 Distributed systems: paradigms and models Motivations Prof. Marco Danelutto Dept. Computer Science University of Pisa Master Degree (Laurea Magistrale) in Computer Science and Networking Academic Year

2 Contents Hardware motivations CPU evolution HPC Clouds Software motivations innovative paradigms can be moved to different frameworks 2

3 Moore s law Moore's original statement can be found in his publication "Cramming more components onto integrated circuits", Electronics Magazine 19 April 1965: The complexity for minimum component costs has increased at a rate of roughly a factor of two per year... Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years. That means by 1975, the number of components per integrated circuit for minimum cost will be 65,000. I believe that such a large circuit can be built on a single wafer. 3

4 Moore s law evolution Transistors/gates doubling every 2 years more and more powerful single processor systems Cores doubling every two years simpler cores more complex (?) memory hierarchy more complex interconnection structure 4

5 Why? doubling core exploits existing technology (and trends) keeping reasonable power consumption doubling the frequency of a single core chip costs much more than putting two simpler cores on the same chip Perf = Freq x IPC Power = DynamicCapacitance x Volt x Volt x Freq ( 5

6 Commodity processors 6

7 Intel perspective 7

8 Intel perspective (2) 8

9 Commodity processors: non Intel 9

10 Commodity processors: niche products 10

11 Research processors: Intel 80 cores 11

12 More in detail... 4Ghz chip with mesh (logical and physical) 10x8 core FP, 1,28 TFlops Tile: router: addesses each core on chip, implements the mesh VLIW processor (96 bit x instruction, up to 8 ops per cycle), in-order-execution, 32 registers (6Read/4Write), 2K Data, 3K Instruction cache, 2 FPU (9 stages, 2FLOPs/cycle sustained), Cicli: FPU:9, Ld/St:2, Snd/ Rcv:2, Jmp/Br:1 12

13 13

14 13

15 14

16 15

17 16

18 GPUs / FPGAs 17

19 Intel Larrabee 18

20 Not only processors: FPGAs q1_embedded_xilinx.htm&usg= PXXvIQmng-24QwOWFUFfFuf1lS4=&h=380&w=650&sz=71&hl=en&start=6&um=1&tbnid=LaX1pZKYodDqSM:&tbnh=80&tbnw=137&prev=/images%3Fq%3Dprocessor %2Bevolution%26hl%3Den%26client%3Dsafari%26rls%3Den%26sa%3DN%26um%3D1 19

21 Not only processors: FPGAs q1_embedded_xilinx.htm&usg= PXXvIQmng-24QwOWFUFfFuf1lS4=&h=380&w=650&sz=71&hl=en&start=6&um=1&tbnid=LaX1pZKYodDqSM:&tbnh=80&tbnw=137&prev=/images%3Fq%3Dprocessor %2Bevolution%26hl%3Den%26client%3Dsafari%26rls%3Den%26sa%3DN%26um%3D1 19

22 Consequence: programming model Heterogeneous computing coming to the scene more and more adaptivity required in the code more and more special purpose solutions needed (transparent to the user) 20

23 Energy concerns/tradeoffs

24 Energy concerns/tradeoff 22

25 Consequence: programming model Faster single core systems faster dusty deck code Multi-many core require parallel / distributed code UMA NUMA 23

26 But... Amdhal law is still there serial fraction = f (% of code not parallelizable) p processors available to parallelize the non serial fraction (1-f) Speedup(p) = Ts / (f Ts + (1-f) (Ts / p)) = 1 / (f + (1-f)/p) asymptotically (when p increases): Speedup(p) = 1 / f 24

27 25

28 HPC evolution: Twice per year top 500 installations measured on standard benchmarks mostly installations from government, military, education, companies Significantly reflecting tendencies kind of Formula 1 in the parallel computing scenario e.g. interconnection networks scaled down to small COW/NOWs 26

29 Top 500: processor family 27

30 Top 500: processor family 27

31 Top 500: processor family 27

32 Top 500: operating system 28

33 Top 500: operating system 28

34 Top 500: operating system 28

35 Top 500: Interconnection network 29

36 Top 500: Interconnection network 29

37 Top 500: Interconnection network 29

38 Top 500: number of processors 30

39 Top 500: number of processors 30

40 Top 500: number of processors 30

41 Moore s law in HPC The Sourcebook of Parallel Computing, Dongarra, Foster, Fox, Gropp, Kennedy, Torczon, White editors,

42 Consequence: programming model Top parallel computing moving towards COW/NOW with smaller and smaller latencies and larger and larger bandwidth 32

43 Evolution in the user model Single processor standard superpipeline superscalar Multi processor ( 70 80) multi/many core ( 00) NOW COW ( 80 90) distributed architecture SSI GRID (late 90 00) meta computing grid (middleware) 33

44 Cloud 34

45 Cloud 35

46 Amazon cloud 36

47 Amazon cloud 37

48 Amazon cloud 37

49 Amazon cloud 38

50 Consequences More and more general architecture virtualization (host, network, operating system,...) Need to adapt to the unknown heterogeneity in hw resources (computing, networking) 39

51 Software evolution Innovative concepts algorithmic skeletons, design patterns, coordination/ orchestration patterns/constructs all introduce efficiency/programmability/... at the price of limitations to programmer freedom software components extreme modular programming (interoperability, commodity and legacy code, portability (w.r.t. framework) services full decoupling usage and implementation 40

52 Software evolution: structured programming Skeletons mostly from HPC community Design patterns mostly from sw engineering community Different approaches language/library vs. programming methodology Different impact Successfully being moved to grids (clouds?) and distributed architectures in general 41

53 Software evolution: components and services Components mainly from sw engineering community (with HPC influences) Services mainly from the business/end user community Different approaches recently merged into a common framework (SCA by IBM et al.) Different impact SOA is everywhere (SaS SOA, IaaS clouds,...) 42

54 Parallel vs. distributed computing McDaniel, George, ed. IBM Dictionary of Computing. New York, NY: McGraw-Hill, Inc., Parallel computing a computer system in which interconnected processors perform concurrent or simultaneous execution of two or more processes Institute of Electrical and Electronics Engineers. IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York, NY: 1990 Distributed computing a computer system in which several interconnected computers share the computing tasks assigned to the system Tanembaum Distributed systems: principles and paradigms, 2nd edition, 2006 Distributed system: a collection of independent computers presenting to the user a single, coherent system image 43

55 Distributed vs. parallel computing 44

56 Distributed vs. parallel computing 45

57 Distributed vs. parallel computing Distributed computing Parallel computing 45

58 Have a look at standard books indexes... Tanembaum, Van Steen Distributed systems: principles and paradigms, 2nd edition, 2006 Introduction, Architectures, Processes, Communications, Naming, Synchronization, Consistency & replicas, Fault tolerance, Security, OO distributed systems, Distributed file system, Web distributed systems, Coordination based systems Kshemkalyani, Singhal Distributed computing: Principles, algorithms and Systems, 2008 Introduction, A model of distributed computations, Logical time, Global state and snapshot recording algorithms, Terminology and basic algorithms, Message ordering and group communication, Termination detection, Reasoning with knowledge, Distributed mutual exclusion algorithms, Deadlock detection in distributed systems, Global predicate detection, Distributed shared memory, Checkpointing and rollback recovery, Consensus and agreement algorithms, Failure detectors, Authentication in distributed systems, Self-stabilization, Peer-to-peer computing and overlay graphs. 46

59 Have a look at standard books indexes... Tanembaum, Van Steen Distributed systems: principles and paradigms, 2nd edition, 2006 Introduction, Architectures, Processes, Communications, Naming, Synchronization, Consistency & replicas, Fault tolerance, Security, OO distributed systems, Distributed file system, Web distributed systems, Coordination based systems Kshemkalyani, Singhal Distributed computing: Principles, algorithms and Systems, 2008 Introduction, A model of distributed computations, Logical time, Global state and snapshot recording algorithms, Terminology and basic algorithms, Message ordering and group communication, Termination detection, Reasoning with knowledge, Distributed mutual exclusion algorithms, Deadlock detection in distributed systems, Global predicate detection, Distributed shared memory, Checkpointing and rollback recovery, Consensus and agreement algorithms, Failure detectors, Authentication in distributed systems, Self-stabilization, Peer-to-peer computing and overlay graphs. Distributed computing 46

60 books... Grama, Gupta, Karypis, Kumar Introduction to parallel computing, 2nd edition 2003 Introduction, Parallel programming platforms, Principles of Parallel Algorithmic design, Basic communication operations, Analytical models of Parallel Programs, Programming using Message passing paradigm, Programming shared address space platforms, Dense matrix algorithms, Sorting, Graph algorithms, Search algorithms for discrete optimization problems, Dynamic programming, Fast Fourier Transform, Appendix: Complexity functions and order analysis Wilkinson, Allen Parallel programming: technique and applications using networks workstations and parallel computers, 2nd edition, 2005 PART I: BASIC TECHNIQUES Parallel computers, Message passing computing, Embarrassingly parallel computations, Partitioning and divide-and--conquer strategies, Pipelined computations, Synchronous computations, Load balancing and termination detection, Programming with shared memory, Distributed shared memory systems and programming PART II: ALGORITHMS AND APPLICATIONS Sorting algorithms, Numerical algorithms, Image processing, Searching and optimization APPENDIXES: Basic MPI routines, Basic Pthread routines, OpenMP directives, library functions and environment variables 47

61 books... Grama, Gupta, Karypis, Kumar Introduction to parallel computing, 2nd edition 2003 Introduction, Parallel programming platforms, Principles of Parallel Algorithmic design, Basic communication operations, Analytical models of Parallel Programs, Programming using Message passing paradigm, Programming shared address space platforms, Dense matrix algorithms, Sorting, Graph algorithms, Search algorithms for discrete optimization problems, Dynamic programming, Fast Fourier Transform, Appendix: Complexity functions and order analysis Wilkinson, Allen Parallel programming: technique and applications using networks workstations and parallel computers, 2nd edition, 2005 PART I: BASIC TECHNIQUES Parallel computers, Message passing computing, Embarrassingly parallel computations, Partitioning and divide-and--conquer strategies, Pipelined computations, Synchronous computations, Load balancing and termination detection, Programming with shared memory, Distributed shared memory systems and programming PART II: ALGORITHMS AND APPLICATIONS Sorting algorithms, Numerical algorithms, Image processing, Searching and optimization APPENDIXES: Basic MPI routines, Basic Pthread routines, OpenMP directives, library functions and environment variables Parallel computing 47

62 Distributed systems: paradigms and models Distributed as a kind of summary word for distributed & parallel Systems systems as a whole : hardware + software Paradigms sample paradigms proven successful to exploit parallel & distributed systems Models programming models to exploit parallel & distributed systems 48

63 Methodology Analysis look for possibilities to apply known techniques/patterns figure out performances Implementation pick up proper tools/mechanisms/models if needed build your own ad-hoc tools Debugging/Tuning rely on application structure Porting rely on tools 49