Module 5 Introduction to Parallel Processing Systems

Transcription

1 Module 5 Introduction to Parallel Processing Systems 1. What is the difference between pipelining and parallelism? In general, parallelism is simply multiple operations being done at the same time.this can be achieved using mutliprocessors, coding in a way that at a time different parts of program could be executed simultaneously..for example threading etc. Pipelining is a particular form of parallelism. Pipelining is a particular arrangement of functions so that different portions of an operation flow through a particular set of sub-functions, with the sub-functions happening in parallel. For example, you have an equation a=b*c+d*e+f*g. step 1: computer shall multiply b*c(multiplier) and forward it to adder. step 2: while previous calculation now is added to zero (adder)...d*e is being calculated in multiplier which will then forward it to adder. Thus, various operations are pipelined so as to work parallely! 2. What is pipelining? Explain with suitable example the difference between serial instruction execution and pipelined instruction execution. Also prove that why pipelined instruction execution is faster as compared to serial instruction execution. 3. What is pipelining? Show that K stage pipelined processor has K times speed up compared to a non pipelined systems. Pipelining is used by virtually all modern microprocessors to enhance performance by overlapping the execution of instructions. If the stages of a pipeline are not balanced and one stage is slower than another, the entire throughput of the pipeline is affected. In terms of a pipeline within a CPU, each instruction is broken up into different stages. Ideally if each stage is balanced (all stages are ready to start at the same time and take an equal amount of time to execute.) the time taken per instruction (pipelined) is defined as: Time per instruction (unpipelined) / Number of stages 10

2 Module 5 : Introduction to Parallel Processing System The previous expression is ideal. We will see later that there are many ways in which a pipeline cannot function in a perfectly balanced fashion. In terms of a CPU, the implementation of pipelining has the effect of reducing the average instruction time, therefore reducing the average CPI. EX: If each instruction in a microprocessor takes 5 clock cycles (unpipelined) and we have a 4 stage pipeline, the ideal average CPI with the pipeline will be Pipelining increases throughput, but not latency Answer available every 200ps, BUT A single computation still takes 1ns Limitations: Computations must be divisible into stage size Pipeline registers add overhead 11

3 4. What is the difference between pipelining and parallelism? Parallel computing is the simultaneous execution of the same task (split up and specially adapted) on multiple processors in order to obtain results faster. The idea is based on the fact that the process of solving a problem usually can be divided into smaller tasks, which may be carried out simultaneously with some coordination. A parallel computing system is a computer with more than one processor for parallel processing. In the past, each processor of a multiprocessing system always came in its own processor packaging, but recentlyintroduced multicore processors contain multiple logical processors in a single package. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors (known as "processing elements" or PEs) and memory. Pipelining is a method of increasing system performance and throughput. It takes advantage of the inherent parallelism in instructions. Instructions are divided into 5 stages : IF, ID, EX, MEM, WB. In pipelining, we try to execute 2 or more instructions at the same time thereby increasing the throughput. With regards to Silicon area and speed, Pipelining takes up more Silicon area. This is because in hardware pipelining we duplicate the few units. In terms of speed, pipelining increases system performance and speed. 5. Explain the Flynn's Classification in detail? Flynn s classifications A taxonomy first introduced by Flynn is still the most common way of categorizing systems with parallel processing capability. Flynn proposed the following categories of computer systems: Single instruction, single data (SISD) stream: A single processor executes a single instruction stream to operate on data stored in a single memory. uniprocessors fall into this category. Single instruction, multiple data (SIMD) stream: A single machine instruction controls the simultaneous execution of a number of processing elements on a lockstep basis. Each processing element has an associated data memory, so that each instruction is executed on a different set of data by the different processors. Vector and array processors fall into this category. 12

4 Module 5 : Introduction to Parallel Processing System Multiple instruction, single data (MISD) stream: A sequence of data is transmitted to a set of processors, each of which executes a different instruction sequence. This structure is not commercially implemented. Multiple instruction, multiple data (MIMD) stream: A set of processors simultaneously execute different instruction sequences on different data sets. SMPs, clusters, and NUMA systems fit into this category. Fig. A taxonomy of Parallel Processor Architectures With the MIMD organization, the processors are general purpose; each is able to process all of the instructions necessary to perform the appropriate data transformation. MIMDs can be further subdivided by the means in which the processors communicate (fig.6.1). If the processors share a common memory, and processors communicate with each other via that memory. The most common form of such system is known as a symmetric Multiprocessor (SMP). In an SMP, multiple processors share a single memory or pool of memory by means of a shared bus or other interconnection mechanism: a distinguishing feature is that the memory access time to any region of memory is approximately the same for each processor. A more recent development is the nonuniform memory access (NUMA) organization. As the name suggests, the memory access time to different regions of memory may differ for a NUMA processor. SISD SISD machines executes a single instruction on individual data values using a single processor. Even if the processor incorporates internal parallelism, such as an instruction pipeline, the computer would still be classified as SISD Uniprocessors 13

5 Fig (a) SISD SIMD As its name implies, an SIMD machine executes a single instruction on multiple data values simultaneously using many processors. Since there is only one instruction, each processor does not have to fetch and decode each instruction. Instead a single control unit handles this task for all processors within the SIMD computer Examples: Illiac-IV, CM-2 Simple programming model Low overhead Flexibility All custom integrated circuits Fig 5.2 SIMD (with distributed memory) MISD: This classification is not practical to implement. So, no significant MISD computers have ever been built. It is included for completeness of the classification. MIMD: Systems referred to as multiprocessors or multicomputer are usually MIMD. It may execute multiple instructions simultaneously, unlike SIMD. So, each processor must include its own control unit. MIMD machines are well suited for general purpose use. Examples: Sun Enterprise 5000, Cray T3D, SGI Origin Flexible 14

6 Module 5 : Introduction to Parallel Processing System Fig. MIMD (with distributed memory) 6. Comparison of pipeline processor and array processor SIMD MIMD 1. It I also called as Array Processor It is also called as Multiprocessor 2. Here, single stream of instruction is fetched Here multiple stream of instructions are fetched 3. In SIMD (single instruction multiple data stream) the instruction stream is fetched by shared memory. In MIMD (multiple instruction multiple data stream) the instruction stream is fetched by control unit. 4. Here instruction is broadcasted to multiple processing elements (PEs), which will operate on different data sets (i.e. data streams are taken from shared memory modules). Hence the name SIMD. Here instruction streams are decoded to get multiple decoded instruction streams (IS), which operate on multiple data stream (DS) taken from shared memory modules. 7. Comparison of loosely coupled and tightly coupled multiprocessor organizations Loosely coupled Multiprocessor 1. In loosely coupled multiprocessor organization communication between computer modules has taken place through message Transfer system Tightly coupled Multiprocessor In tightly coupled multiprocessor organization communication between processor s can be taken place through P MIN. 15

7 2. Diagram is as follows I S I N Disks R LM CA S I D R LM CA S I D P 1 P n P M I N m I O P N 1 Unmapped local Memory (stores kernel) Memory map Message Transfer System (Time Shared BUS) (MTS) 3. In loosely coupled system, the processor do not share memory, and each processor has its own local memory. M 1 M 2 M 3 Shared memory modules In, tight coupled system, there is a single system-wise primary memory that is shared by all the processors. LM CPU LM CPU LM CPU LM CPU CPU CPU System-wise LM shared CPU memory CP U Communication N/W Interconnection hardware 4. In these systems, all physical communication between the processors is done by passing message across the network that interconnects the processors. 5. Loosely coupled systems, are referred as distributed computing Systems. 6. In this system, the interaction between various processors and computer modules is very less. 7. In this, (As channel and Arbiter Switch) is used to connect the computer module to a message transfer system. In these systems, communication between the processors usually takes place through the shared memory. In this system, the interaction between various processors is very high. In this system, the interaction between various processors is very high. In this, shared memory modules can communicate through PMIN. 16

8 Module 5 : Introduction to Parallel Processing System 8. In this, the processor is directly connected to IO devices. 9. Distributed memory multiprocessors are sometimes referred as loosely coupled system. 10. There are no unmapped local memories in loosely coupled system. In this, processor connected to IO device through IOPIN. Shared memory multiprocessor are sometimes referred as tightly coupled system. In this tightly coupled system, every processor has small unmapped local memories which is used to store Kernel code. 8. What is instruction pipelining? Instruction processing is subdivided into Fetch/ Execute instruction Pipeline has two independent stages: 1 st Stage Fetch an instruction and buffers it. 2 nd Stage Temporarily free until first stage passes it the buffered instruction. While the second stage is executing the instruction, the first stage fetches and buffers the next instruction. Instruction prefetch or fetch overlap. 17

9 Clock Cycle Instruction I1 F1 E1 I2 F2 E2 I3 F3 E3 9. What is 5 stage instruction pipelining?(risc Pipeline) The five clock cycles will be broken up into the following steps: Instruction Fetch Cycle Instruction Decode/Register Fetch Cycle Execution Cycle Memory Access Cycle Write-Back Cycle When a CPU executes an instruction, it transitions through number of stages to complete. These basic stages (or phases) are: instruction fetch, instruction decode, register fetch, execution/effective-address computation, memory access and write back to registers. Instruction fetch (IF) use the current program counter (PC) to index into memory and fetch the instruction. The act of fetching from memory may cause an exception and other complicated interactions with the hardware memory optimizations. Instruction decode/register fetch (ID) this phase combines decoding of the next instruction with the latching of the current register values from the register file with the interpretation of the operation to be executed. A suitably designed instruction-set architecture it is possible to determine the 18

10 Module 5 : Introduction to Parallel Processing System registers operated on by the command independently of what the actual instruction is, which allows the register fetch to occur in parallel with the instruction decode. Execution/effective-address computation (EXE) the arithmetic logic unit (ALU) is central to instruction execution. Given the decoded operation and latched operands, the ALU performs the required operation, which can be either register arithmetic or an effective-address computation. The result is then latched for the next phase of execution. Memory access (MEM) either store a latched operand into memory indexed by the effective-address(ea) computed in the previous phase or load and latch a value from the computed EA. Write back to registers (WBR) Transfer the latched output of the memory access or execution phases to the required register. Some instructions do not need a WBR phase. A simple CPU implementation would perform each phase in a clock cycle and every instruction would take up to 5 clock cycles to complete. Not all instructions require all phases, for instance a simple register indirect store instruction does not need to perform the execution or the write back phases and can complete in 3 cycles. Early computer designers realized that this implementation leaves most of the CPU idle for most of the time and does not use the resources efficiently. In 1960 the designers of the IBM-7030 (Stretch because it was supposed to be a stretch in the state of the art) introduced the concept of an instruction pipeline. As figure 6.3 below shows, a pipeline can overlap the execution phases of an instruction with previous instructions. 10. Explain Four stage CPU instruction pipeline. 19

11 Explain Six stage CPU instruction pipeline

12 Module 5 : Introduction to Parallel Processing System Advantages Multiple instructions are being processed at same time This works because stages are isolated by registers Best case speedup of N Certain preventions Instructions interfere with each other - hazards Example: different instructions may need the same piece of hardware (e.g., memory) in same clock cycle 21

13 Example: instruction may require a result produced by an earlier instruction that is not yet complete 12. Explain various types of hazards in pipelined processors with example. Also propose solution each type. 22

14 Module 5 : Introduction to Parallel Processing System 23

15 24

16 Module 5 : Introduction to Parallel Processing System 25

17 26

18 Module 5 : Introduction to Parallel Processing System 27

19 13. Explain Multiprocessor architectures. Multiprocessor architecture : This is nothing but MIMD architecture. These are classified into two categories : i) Loosely coupled multiprocessor 28

20 ii) Closely (tightly) coupled multiprocessor. Module 5 : Introduction to Parallel Processing System Loosely Coupled multiprocessor : This system contains a number of computer modules each containing a processor, local memory and I/O devices. A switch called as CAS is used to contact the computer module to a message transfer system (MTS) which is generally a time shared bus. It is used for message transfer between two computer modules. The diagram is as follows : Computer Module 1 Computer Module N LM IO L M I O P P CAS CA S Message Transfer System (MTS) (Time Shared Bus) Fig : Loosely coupled multiprocessor P : Processor LM : Local Memory IO : I/O devices CAS : Channel and arbiter Switch Here work load is distributed among various computer modules. One computer module and multiprocessor operating system is responsible for distribution of the work load. Each of that computer will carry out the assigned work local in a distributed / independent fashion. Hence it is sometimes called a distributed system. There are vary less amount of interaction between various processors / computer modules through time shared bus, hence it is called as Loosely Coupled System. The performance of loosely coupled system (LCS) depends on following factors : 1. Bandwidth of time shared bus which defines the speed of message transfer. 29

21 30 2. Message length 3. Message arrival rate, which define the degree of interaction / coupling between various computer modulus. Symmetric Multiprocessors An SMP can be defined as a standalone computer system belonging to tightly coupled multiprocessing system and has the following characteristics: 1. There are two or more similar processors of comparable capability 2. These processors share the sane main memory and I/O facilities and are interconnected by a bus or other internal connection scheme, such that memory access time is approximately the same for each processor. 3. All processors share access to I/O devices, either through the same channels or through different channels that provide paths to the same device. 4. All processors can perform the same functions (hence the term symmetric) 5. The system is controlled by an integrated operating system that provides interaction between processors and their programs at the job, task, file, and data element levels. Clusters Clustering is an alternative to symmetric multiprocessing as an approach to providing high performance and high availability and is particularly attractive for server applications. Here the physical unit of interaction is usually a message or complete file. Whereas in SMP, individual data elements can constitute the level of interaction, and there can be a high degree of cooperation between processes. Clusters belong to loosely coupled multiprocessing system. A cluster can be defined as a group of interconnected, whole computers working together as a unified computing resource that can create the illusion of being one machine. The term whole computer means a system that can run on its own, apart from the cluster. Nonuniform Memory Access (NUMA) Here, all processors have access to all parts of main memory using loads and stores. The memory access time of a processor differs depending on which region of main memory is accessed. The last is true for all processors; however, for different processors, which memory regions are slower and which are faster differ. NUMA belongs to tightly coupled multiple-processor system. A special kind of NUMA is CC-NUMA (Cache-coherent NUMA) in which cache coherence is maintained among the caches of the various processors.