Message Passing Models and Multicomputer distributed system LECTURE 7

Transcription

1 Message Passing Models and Multicomputer distributed system LECTURE 7 DR SAMMAN H AMEEN 1

2 Node Node Node Node Node Node Message-passing direct network interconnection Node Node Node Node Node Node PAGE 2

3 Computers that rely on message passing for communication rather than cache coherent shared memory are much easier for hardware designers to build There is an advantage for programmers as well, in that communication is explicit, which means there are fewer performance surprises than with the implicit communication The downside for programmers is that it's harder to port a sequential program to a message- passing computer, since every communication must be identified in advance or the program doesn't work Cache-coherent shared memory allows the hardware to figure out what data needs to be communicated, which makes porting easier There are differences of opinion as to which is the shortest path to high performance, given the pros and cons of implicit communication, but there is no confusion in the marketplace today Multicore microprocessors use shared physical memory and nodes of a cluster communicate with each other using message passing PAGE 3

4 Advantages Easier to build than scalable shared memory machines Easy to scale (but topology is important) Programming model more removed from basic hardware operations Coherency and synchronization is the responsibility of the user, so the system designer need not worry about them Disadvantages Large overhead: copying of buffers requires large data transfers (this will kill the benefits of multiprocessing, if not kept to a minimum) Programming is more difficult Blocking nature of SEND/RECEIVE can cause increased latency and deadlock issues PAGE 4

5 Basic communication constructs/ primitives Send(destination, message) Receive(source, message) Here, source/destination is process name or link mailbox or port Issues in message passing communication Direct Vs Indirect Blocking Vs Non blocking Reliable Vs Unreliable Buffered Vs Unbuffered PAGE 5

6 Unique process name This can be done by concatenating host name and process id to identify the source/destination Note that the format for both send and receive primitives are symmetric There is a bidirectional communication between source and destination That means, for every send primitive there is a receive primitive A B PAGE 6

7 buffer[]) buff[]) Format is Symmetric, Communication is bidirectional (for every Send/Recv, there is Recv/Send command) PAGE 7

8 Receive message from unknown source Source in Receive primitive is a variable which gets the value (sender id) when the message is received Asymmetric sender and receive primitive [Only sender need to specify the receiver id and receiver may receive the message from any sender] Unidirectional path between sender and receiver [No waiting for specific sender by receiver] A C B PAGE 8

9 variable = p1@machine1, on receipt of message from p1@machine1 variable = p2@machine2 on receipt of message from p2@machine2 variable = p3@machine3 on receipt of message from p3@machine3 PAGE 9

10 This type of communication is used when sending processes are not concerned with the identity of receiving processes and receiving processes are interested only in the message and not the identity of sending processes Example: Multiple clients may request services from one of the multiple servers Network Any client request may be served by any available server So sender (client) and receiver (server) concerned only with data and not id of each other Clients Servers PAGE 10

11 Process Kernel Network When sending the message from source to destination process, the message is passed to sender kernel, which is then pass over communication network, then to receiver kernel; finally to the destination process PAGE 11

12 While sending the message, synchronization can take place at one or more of the following points Between Sender and Sending kernel Between Sending kernel and Receiving kernel Between Receiving kernel and Receiver Based on above synchronization points, we have different kind of semantics under send and receive primitives Buffering space is present in the sender s kernel, receiver s kernel and in the communication network These can be logically combined into a single buffer PAGE 12

13 Blocking receive: RECEIVE Receive has only one semantic, that is the process has to wait till a message is received by the process S e n d Recv() PAGE 13

14 Non blocking: The sender process is released after the message has been composed and copied onto sender s kernel This is asynchronous send SEND Buffer space is assumed to be unbound The sender process is blocked if the sending kernel is not ready to accept the message PAGE 14

15 Blocking send: SEND Sender process is released after the message has been transmitted into the communication network Reliable blocking send: Sender process is released after the message has been received by the destination kernel Explicit blocking send: Sender process is released after the message has been received by the destination (receiver) process Request and reply: Sender process is released after the message has been received by the receiver and response is returned to the sender Also called Client Server communication PAGE 15

16 SEND Sending Process Receiving Process Sending Kernel Network Receiving Kernel SENDER IS BLOCKED PAGE 16

17 SEND Sending Process Receiving Process Sending Kernel Receiving Kernel Network SENDER IS BLOCKED NON BLOCKING SEND (till message reaches S kernel) PAGE 17

18 SEND Sending Process Receiving Process Sending Kernel Receiving Kernel Network SENDER IS BLOCKED BLOCKING SEND (till message reaches n/w) PAGE 18

19 SEND Sending Process Receiving Process Sending Kernel Receiving Kernel Network SENDER IS BLOCKED RELIABLE BLOCKING SEND (till message reaches R kernel) PAGE 19

20 SEND Sending Process Receiving Process Sending Kernel Network Receiving Kernel SENDER IS BLOCKED EXPLICIT BLOCKING SEND (till message reaches receiver) PAGE 20

21 SEND Sending Process Sending Process Receiving Process Receiving Process Sending Kernel Sendi ng Receiving Kernel Receiving Kernel Network Kernel Network SENDER IS BLOCKED EXPLICIT BLOCKING SEND (till SENDER message IS BLOCKED reaches receiver) NON BLOCKING RELIABLE EXPLICIT REQUEST BLOCKING AND SEND REPLY SEND (till SEND message (till (till response (till message reaches message from reaches n/w) reaches receiver S kernel) receiver) Rreaches kernel) back sender) PAGE 21

22 It is the communication endpoint of communication link and is managed by the transport services Sockets are used for communication between processes in heterogeneous domain It is established by socket system call, which returns a descriptor and informs the kernel about the protocol (TCP/UDP) the process is going to use The system call is described by (protocol, local address, local process/port, foreign address, foreign process/port) PAGE 22

23 SERVER CLIENT Protocol Socket Socket Protocol Local address + port Bind() Connect() To the Local address + port given by the system Ready to accept connection Listen() Block until connection from client Accept() Read() Data (Request) Write() Process request Write() Data (Reply) Read() PAGE 23

24 Protocol SERVER Socket CLIENT Socket Block until Bind() Bind() data received from the client Receive From() Send To() Send To() Receive From() PAGE 24

25 Idea is group of processes receive messages from source If group size = N, communication is broadcasting Else (group size < N) communication is multicasting N : number of processes in a DS The source may belong to the same group of recipients The source may or may not belong to the same group of recipients PAGE 25

26 Unanimity: A message is delivered either to all processes in the group or not at all (atomic property ) Termination The result of a message submission to the broadcast server is known to the submitted broadcast server within a finite time period Uniformity: If a message is delivered to a process in a group, then it should be delivered to all the processes in that group Order: BS will deliver the messages to APs in the group based on some predefined ordering requirements PAGE 26

27 Wide area network (WAN); A WAN connects a large number of computers that are spread over large geographic distances It can span sites in multiple cities, countries, and continents Metropolitan area network (MAN); The MAN is an intermediate level between the LAN and WAN and can perhaps span a single city Local area network (LAN); A LAN connects a small number of computers in a small area within a building or campus System or storage area network (SAN) A SAN connects computers or storage devices to make a single system PAGE 27

28 A network channel c=(x,y) is characterized by width wc: the number of parallel signals it contains, frequency fc: the rate at which bits are transported at each signal latency tc is the time required for a bit to travel from x to y A bandwidth of a channel is W= wc * fc The throughput Θ of a network is the data rate in bits per second that network accepts per input port Under a particular traffic pattern, the channel that carries the largest fraction of the traffic determines the maximum channel load γ Load on the channel can be equal or smaller than channel bandwidth Θ=W/γ PAGE 28

29 Deterministic: The simplest algorithm - for each source, destination pair, there is a single path This routing algorithm usually achieves poor performance because it fails to use alternative routes, and concentrates traffic on only one set of channels Oblivious: So named because it ignores the state of the network when determining a path Unlike deterministic, it considers a set of paths from a source to a destination, and chooses between them Adaptive: The routing algorithm changes based on the state of the network PAGE 29

30 Message: logical unit for internode communication Packet: basic unit containing destination address for routing Packets have sequencing # for reassembly Flits: flow control digits of packets PAGE 30

31 Header flits contain routing information and sequence number Flit ( FLow control digits) length affected by network size Packet length determined by routing scheme and network implementation Lengths also dependent on channel b/w, router design, network traffic, etc PAGE 31

32 100 Gigabit Ethernet (100GbE) and 40 Gigabit Ethernet (40GbE) are groups of computer networking technologies for transmitting Ethernet frames at rates of 100 and 40 gigabits per second (100 and 40 Gbit/s), respectively The technology was first defined by the IEEE 8023ba-2010 standard InfiniBand (abbreviated IB) is a computer network communications link used in high-performance computing featuring very high throughput and very low latency It is used for data interconnect both among and within computers As of 2014 it is the most commonly used interconnect in supercomputers PAGE 32

33 Latency is an element that contributes to network speed The term latency refers to any kind of delay typically incurred in processing of network data A low latency connection is one that generally experiences small delay times, while a high latency connection generally suffers from long delays PAGE 33

34 In the context of parallel computing, granularity is the ratio of communication time over computation time Fine grain parallelism is characterized by seemingly more communications as the relative computation time is shorter Coarse grain parallelism, then, is characterized by seemingly fewer communications with much longer computation time Load balance is easier to achieve with fine grain parallelism because small tasks depend less on the operating system, interrupts and so on Coarse grain parallelism, on the converse, makes it harder to predict when any given task will terminate, therefore making it harder to assign tasks for optimal usage of the multiple processors Fine grain parallelism requires more synchronization overhead due to the need to communicate data and synchronize tasks among processors Therefore, the fewer communications in coarse grain parallelism reduces overhead PAGE 34