Instruction Level Parallelism (ILP)

Transcription

1 Instruction Level Parallelism (ILP) Pipelining supports a limited sense of ILP e.g. overlapped instructions, out of order completion and issue, bypass logic, etc. Remember Pipeline CPI = Ideal Pipeline CPI + Struct. Stalls + RAW Stalls + WAR Stalls + WAW Stalls + Control Stalls Issues involving crossing branches Now turn consideration to methods which reduce all of the terms on the RHS of the equation. solution is of course a combo of HW and SW/Compiler methods Chapter 4 page 1

2 Methods Table 1: Technique Loop Unrolling Basic Pipeline Scheduling (we ve seen some of this already) Dynamic Scheduling with Scoreboarding Dynamic Scheduling with register renaming Dynamic Branch Prediction Issuing multiple instructions per cycle a.k.a (also known as) Superscalar Compiler dependence analysis Software pipelining and trace scheduling Speculation Dynamic memory dependency analysis Reduces Control Stalls RAW stalls RAW stalls RAW stalls Control Stalls Ideal CPI Ideal CPI and data stalls (RAW, WAW, WAR) Ideal CPI and data stalls data and control stalls RAW stalls involving memory Chapter 4 page 2

3 ILP within a Basic Block Basic Block definition straightline code - no branches out single entry point at top ==> real code is a bunch of basic blocks connected by branch statements Notice branch frequency is approx. 15% of total mix implies basic block size between 6 and 7 instructions these 6-7 instructions are likely to depend on one another there s probably little to exploit in ILP Easiest target is the loop exploiting loop-level parallelism, i.e., among iterations of a loop because every iteration can overlap with any other iteration Chapter 4 page 3

4 Loop Level Parallelism Consider adding two 1000 element arrays for (i=1, i<=1000, i=i+1) x[i] = x[i] + y[i]; Sure it s trivial but it illustrates the point there is no dependence between data values produced in any interation j and those needed in j+n for any j and n truly independent - hence could be a 1000-way parallel program independence means no stalls due to hazards problem is that we have to use branch instructions and while prediction is easy - we have to be able to predict accurately Vector Processor model load vectors x and y (up to some machine dependent max) then do result-vec = xvec + yvec in a single instruction Chapter 4 page 4

5 Some Assumptions Isn t this course just full of them - this time they re stated Default DLX pipeline structure Instruction Producing Value Table 1: Instruction Consuming Value Intervening Clock Cycles to Avoid Stalls FP ALU Op FPU ALU Op 3 FP ALU Op Store Double 2 Load Double FP ALU Op 1 Load Double Store Double 0 Integer Load Integer ALU Op 1 Integer ALU Op Integer ALU Op 0 Branch Delay Slots Anything 1 Note: latencies are somewhat smaller than we saw last lecture Reason: make the examples less cumbersome Chapter 4 page 5

6 Loop Unrolling Consider adding a scalar to a vector for (i=1, i<=1000, i++) x[i] = x[i] + s loop: LD ADDD SD SUBI BNEZ FO, 0(R1) F4, F0, F2 0(R1), F4 R1, R1, #8 R1, loop ;R1 array ptr ;F2 = s ;put result ;back ;decr. R1 ;again if not ;done for convenience of exit the array starts at location 0 How does it run with no scheduling 10 cycles per iteration LD, load stall, ADDD, 2 RAW stalls, SD, SUBI, stall, BNEZ branch delay control stall Let s try to fill the delayed branch stall Chapter 4 page 6

7 Note: Scheduling the Loop Non-trivial and many compilers don t try this move SD to branch delay slot but since it will now be after the SUBI we ll need an offset of 8 SUBI doesn t have an RAW hazard with the add so no stall Result will run as: loop: 40% improvement 10 cycles ==> 6 but still a 3-cycle overhead in each loop iteration loop: LD ADDD SD SUBI BNEZ FO, 0(R1) F4, F0, F2 0(R1), F4 R1, R1, #8 R1, loop LD SUBI ADDD stall BNEZ SD FO, 0(R1) R1, R1, #8 F4, F0, F2 R1, loop 8(R1), F4 necessary Chapter 4 page 7

8 Basic Idea Loop Unrolling unroll n loop iterations into 1 basic block adjust the new termination code Let s say n is 4 Then modify the R1 pointer in the example by 4x of what it was before ==> 32 Savings - 4 BNEZ s + 4 SUBI s => just one of each Hence 75% improvement Problem - still have 4 load stalls per loop Next idea don t concatenate the unrolled segments shuffle them instead 4 LD s then 4 ADDD s then 2 SD s, SUBI, SD, BNEZ, then fill the branch delay slot with the final SD No more stalls afterward since LD to dependent ADDD path now has 3 instructions in it. Chapter 4 page 8

9 Loop: Result LD LD LD LD ADDD ADDD ADDD ADDD SD SD SUB1 SD BNEZ SD Result of Unrolling F0, 0(R1) F6, -8(R1) F10, -16(R1) F14, -24(R1) F4, F0, F2 F8, F6, F2 F12, F10, F2 F16, F14, F2 0(R1), F4-8(R1), F8 R1,R1,#32 16(R1) R1, Loop 8(R1), F16 Note: still didn t run out of registers - so we could have improved on this = -24 must compensate for final SD being after the SUBI 14 cycles for 4 loop iterations==> 3.5 cycles per iteration 10 cycles per iteration with no scheduling 6 cycles per iteration with scheduling but no unrolling Chapter 4 page 9

10 Things to Notice We had 8 more unused register pairs hence we could have gone to an 8 block unroll without register conflict problem is that then the 1000 element array would have to be broken cleanly (1000/8 = 125) what if it had not - say with remainder R so what -- just put R blocks in front of or outside of the loop and run you ll notice that you run out of registers but by cycling through the names you can still remove any stalls in this case Still there were many things the compiler had to notice to figure this one out (trickiest was the SD/SUBI swap) Key was the independent nature of each loop body 3 types of dependence: data, name, and control Chapter 4 page 10

11 Data Dependency Analysis i depends on j if: i uses a result produced by j OR i uses a result produced by k and k depends on j dependence indicates a possible RAW hazard whether it is really there is a property of the pipeline (e.g., OK after 3 cycles) and the structure of the forwarding logic But the possibility controls order in which results must be calculated sets a bound on how much parallelism can be exploited and therefore it affects the performance compilers static dataflow analysis uses a directed graph that makes these dependencies explicit Chapter 4 page 11

12 Occurs when Name Dependence 2 instructions use the same register name BUT no flow of data between the two insts. actually exists Two types of name dependences let i precede j in program order i is antidependent on j when j writes a register that i reads essentially the same as a WAR hazard hence the ordering must be preserved to avoid the hazard i is output dependent on j if they both write to the same register yep - a WAW hazard so we have to avoid that too No real data dependence -- just a name conflict so register can be renamed statically by the compiler or dynamically by the hardware to eliminate name dependences Chapter 4 page 12

13 Control Dependence Since branches are conditional a control-dependent inst. should not execute until branch resolved instructions before the branch don t matter Consider the following code: S1; if P then S2 else S3; 2 obvious constraints instructions controlled by the branch cannot be moved before the branch, e.g., cannot move "S2" ot "S3" before "if P". an instruction not controlled by the branch cannot be moved after the branch, e.g., cannot move S1 after "if P". Chapter 4 page 13

14 More on Control Dependence Violating the constraints? put a control-dependent inst before the branch? OK if we can trash the result when the branch goes the wrong way works when the result goes to a register which becomes dead (result never used) if the wrong way is taken However 2 important side-effects affect correctness issues exception behavior remains intact sometimes this is relaxed but it probably shouldn t be branches effectively set up conditional data flow data flow is definitely real so if we do the move then we better make sure it doesn t change the data flow So it can be done but care must be taken Discuss it in Sections 4.5 and 4.6 on speculation execution Chapter 4 page 14

15 Some Examples Start with the old example for (i=1, i<=1000, i++) x[i] = x[i] + s loop: LD ADDD SD SUBI BNEZ FO, 0(R1) F4, F0, F2 0(R1), F4 R1, R1, 8 R1, loop ;R1 array ptr ;F2 = s ;put result ;back ;decr. R1 ;again if not ;done Name, Control, and Data dependencies? (SUBI produces a value needed in the next loop) SUBI-induced data dependencies with insts. in the next loop are removed by code transformation, i.e., removing intermediate SUBIs Name dependencies are resolved by register renaming Control dependencies are minimized by removing intermediate BRs Compiler can do all these Chapter 4 page 15

16 Implication for (i=1; i<=1000; i++) { A[i+1] = A[i] + C[i]; B[i+1] = B[i] + A[i+1];} Now Consider /* S1*/ /* S2*/ S1 uses a value produced by S1 in an earlier iteration S2 also uses a value produced by S2 in an earlier iteration S2 uses a value produced by S1 in the same iteration S1 depends (loop-carried) on S1 hence their order must be preserved Similar for the S2 carried dependence Note if non-loop-carried dependences were the only ones then the order would not matter Chapter 4 page 16

17 One More for (i=1; i<=100; i++) { A[i] = A[i] + B[i]; B[i+1] = C[i] + D[i];} /* S1*/ /* S2*/ S1 uses a value produced by S2 in an earlier iteration However this loop-carried dependence is not circular since neither statement depends on itself no cycle: S1 depends on S2 but S2 does not depend on S1 Implications No cycle in dependencies ==> loop can be parallelized to: A[1] = A[1] + B[1]; No loop-carried dependence so UNROLL for (i=1; i<=99; i++) { B[i+1] = C[i] + D[i]; /* S1*/ A[i+1] = A[i+1] + B[i+1];} /* S2*/ B[101]=C[100] + D[100]; Chapter 4 page 17

18 Dynamic Pipeline Scheduling Compiler does Static Scheduling Hardware rearranges instruction stream to reduce stalls Treats dependencies which are only known at run time - e.g. memory reference Simplifies the compiler Even permits compiled code to run on a different pipeline Minimizes the need for speculative slot filling Unlike DLX which holds instruction issue on stalls Issues and executes instructions that are ready Chapter 4 page 18

19 Dynamic Scheduling a.k.a. dynamic instruction reordering allow out-of-order issue consider DIVD ADDD SUBD F0, F2, F4 F10, F0, F8 F12, F8, F14 DIVD has a long latency ADDD has a data dependence on F0, SUBD does not so swap them - compiler might have done this but so could the hardware Problems with out-of-order issue? precise interrupts -- we will discuss it in Section 4.6 Chapter 4 page 19

20 Scoreboarding Allowing instructions to be issued out-of-order Instructions ID IF Window Scoreboard Note: normal ID stage now split in 2: 1. issue 2. read operands EX CPU Pipeline WB Chapter 4 page 20

21 DLX Scoreboard Example More Interesting Code Fragment DIVF F0, F2, F4 ADDF F10, F0, F8 RAW hazard (data dependence) SUBF F8, F8, F14 WAR hazard (antidependence) Note following ADDF can t start until DIVF completes SUBF doesn t need to wait but can t post result to F8 until ADDF reads F8 Change the DLX pipeline design ID is split into 2 stages: Issue and Read Operands -- Issue: decode and issue if no structural and WAW hazards -- Read Operands: wait until no data hazard; then read operands Execute: just do it - remember multiple EX units available Chapter 4 page 21

22 Scoreboards Introduced with the CDC FPU s 5 MMU s 7 IU s Centralized hazard detection Load/Store architecture Results approx. 60% due to dynamic ordering cost inflates by approx. 20% still fastest machine of it s day a few years ago technique too complex for single chip designs Now both IBM and HP bring it back into the game Chapter 4 page 22

23 DLX with a Scoreboard Registers FP Mult FP Mult FP Divide FP Add Note: this model could support both single or multi-issue more than one multiple insts. will be issued per cycle Integer Unit One set of buses serves a group of FUs SCOREBOARD Chapter 4 page 23

24 New 4 step Pipeline Section RULES Really 6 but ignore IF & Mem to focus on Reg-Reg issues OLD ID EX WB Issue free execution unit AND no destination register conflict ==> issue else stall e.g. check for structural and WAW hazards (one S.Hz might be Opnd and Result busses) nothing out of order yet - if issue stalls then IF stalls when IF/ISSUE buffer fills Read Operands scoreboard monitors source operand availability available if no earlier active is going to write it OR it is being written by an active dynamic RAW resolution ==> insts. can be sent into execution out of order now Execution Starts when operands are received May take multiple cycles so unit notifies the scoreboard when it s done Write Result Scoreboard checks for WAR hazards before permitting this stage to happen Chapter 4 page 24

25 could have picked better names Scoreboard Data Structure Components Instruction Status indicates which step the instruction is in Functional Unit Status Busy - indicates whether the FU is doing something Op - operation being performed by the FU Fi - destination Register Fj, Fk - left and right source register numbers Qj, Qk - FU producing source values (Qj --> Fj, Qk -->Fk) Register Result Status indicates which active FU will provide a value for each register blank means nothing active will post to this register Note overlap with Fj, Fk, Qj, Qk fields Chapter 4 page 25

26 Example - Remember the RULES Table 1: Instruction Issue RD Opnd Ex. Done Wr. Result Inst. Status LD F6, 34(R2) X X X X LD F2,45(R3) X X X MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10,F0,F6 ADDD F6, F8, F2 Where are the potential hazards? Table 2: X X X Rj & Rk = No when both are read Name Busy Op Fi Fj Fk Qj Qk Rj Rk FU Status Integer Yes load F2 R3 No Mult1 Yes Mult F0 F2 F4 Int. No Yes Mult2 No Add Yes Sub F8 F6 F2 Int. Yes No Divide Yes Div F10 F0 F6 Mult1 No Yes Table 3: RREG F0 F2 F4 F6 F8 F10 F12... F30 FU Mult1 Int. Add Divide Chapter 4 page 26

27 Blank Forms - work out each step Table 1: Instruction Issue RD Opnd Ex. Done Wr. Result Inst. Status LD F6, 34(R2) LD F2,45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10,F0,F6 ADDD F6, F8, F2 Table 2: Name Busy Op Fi Fj Fk Qj Qk Rj Rk FU Status Integer Mult1 Mult2 Add Divide RREG FU Table 3: F0 F2 F4 F6 F8 F10 F12... F30 Chapter 4 page 27

28 Blank Forms - work out each step Table 1: Instruction Issue RD Opnd Ex. Done Wr. Result Inst. Status LD F6, 34(R2) LD F2,45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10,F0,F6 ADDD F6, F8, F2 Table 2: Name Busy Op Fi Fj Fk Qj Qk Rj Rk FU Status Integer Mult1 Mult2 Add Divide RREG FU Table 3: F0 F2 F4 F6 F8 F10 F12... F30 Chapter 4 page 28

29 Blank Forms - work out each step Table 1: Instruction Issue RD Opnd Ex. Done Wr. Result Inst. Status LD F6, 34(R2) LD F2,45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10,F0,F6 ADDD F6, F8, F2 Table 2: Name Busy Op Fi Fj Fk Qj Qk Rj Rk FU Status Integer Mult1 Mult2 Add Divide RREG FU Table 3: F0 F2 F4 F6 F8 F10 F12... F30 Chapter 4 page 29

30 Blank Forms - work out each step Table 1: Instruction Issue RD Opnd Ex. Done Wr. Result Inst. Status LD F6, 34(R2) LD F2,45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10,F0,F6 ADDD F6, F8, F2 Table 2: Name Busy Op Fi Fj Fk Qj Qk Rj Rk FU Status Integer Mult1 Mult2 Add Divide RREG FU Table 3: F0 F2 F4 F6 F8 F10 F12... F30 Chapter 4 page 30

31 For the CDC 6600 Scoreboarding Results 70% improvement for Fortran 150% improvement for hand coded assembly language cost was similar to one of the functional units surprisingly low bulk of cost was in the extra busses Still this was in ancient time no caches no main semiconductor memory no software pipelining not so powerful compilers So why is it coming back? dynamic scheduling exploits more ILP via out-of-order issue Chapter 4 page 31

32 Intrinsic Scoreboard Limitations Amount of ILP mainly limited by true dependencies in a basic block compiler can help reduce dependencies - unrolling, scheduling, etc. Number of Scoreboard Entries meaning how far to look ahead called a window size the larger the better how to handle branches? -- use speculation see Section 4.6 Number and Types of Functional Units more means less structural hazards Antidependecies and Output Dependencies WAR and WAW hazards respectively renaming (use virtual registers) and more registers will help Chapter 4 page 32

33 Tomasulo's Scheduling Algorithm used in IBM 360/91 (3 years after CDC 6600) Attempt to deal with long memory and FP latencies IBM 360/91 permits Reg-Mem instructions and several iterative style compare instructions as well 2 main differences from the CDC scoreboard Hazard detection and interlocks are entirely distributed Each functional unit uses "reservation stations" to control execution reservation stations act independently as interlock permits Results passed directly to functional units rather than through the registers (that is, via virtual registers) forwarding is achived by broadcasting results to functional units viacdb (common data bus) achieved by a combination of "register renaming" and "tagging" mechanisms Basically a dataflow engine - hence no WAR or WAW hazards can exist - use rename instead Chapter 4 page 33

34 Reservation Station Duties Know which functional unit is to produce the desired source result Read sources off CDB when they appear When all operands are present enable the associated functional unit to execute Since values aren t really written to registers: No WAx hazards are possible Structural hazards checked at issue time for operations: this means reservation station availability for loads/stores: this means buffer availability Otherwise stall and try to issue other insts. in the window Chapter 4 page 34

35 The Tomasulo DLX From Memory From Instruction Unit Load Buffers FP Op. Queue FP Registers Store Buffers Reservation Stations FP Adders CDB Common Data Bus FP Multipliers Note: piped or 1 issue per cycle replicated FU s is same concept Chapter 4 page 35

36 Issue Instruction Steps get instruction from FP Queue for adds, mults. etc. issue it if there is an empty reservation station if the operands are in registers send them to the reservation station for loads/stores issue it available buffer is available stall otherwise due to the structural hazard; check other insts. in the queue Execute (by each reservation station independently of others) when all operands are available then execute if not then monitor CDB to grab desired operand when it is produced effectively deals with RAW hazards Write Result (also done by each reservation station) when result is available write it to the CDB from CDB it will go to a waiting reservation station and to the registers note renaming model prevents WAW and WAR hazards as a side effect Virtual registers with "Tags" a result broadcast to CDB is associated with a tag the tag identifies a virtual register ID that corresponds to a reservation station or a load buffer Chapter 4 page 36

37 Tomasulo's Data Structures Each Reservation Station: Op - the operation Qj, Qk - tag of reservation station or load buffer that will produce the operand blank ==> value is already available and copied to Vj, Vk Vj, Vk - the value of the operands (a copy) Busy - reservation station and its corresponding functional unit are occupied Register file and store buffers Qi - tag of reservation producing the store value Qi=blank means the value stored is ready and up-to-date Chapter 4 page 37

38 Example - all instructions issued, only 1st Load completed Instruction Status Table 1: Instruction Issue Execute Write Result LD F6, 34(R2) X X X LD F2, 45(R3) X X MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10, F0, F6 ADDD F6, F8, F2 Hazards? X X X X Reservation Stations Table 2: Note: load and store buffer busy also needed Name Busy Op Vj Vk Qj Qk Add1 Yes SUB Mem[34 + Regs[R2]] Load2 Add2 Yes Add Add1 Load2 Add3 No Mult1 Yes Mult Regs[F4] Load2 Mult2 Yes Div Mem[34 + Regs[R2]] Mult1 0 Register Status Table 3: F0 F2 F4 F6 F8 F10... F30 Mult1 Load2 Add2 Add1 Mult2 Chapter 4 page 38

39 Blank Forms LD1 LD2 LD3 LD4 LD5 LD6 Busy Addr Load Status LD F6, 34(R2) LD F2, 45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10, F0, F6 ADDD F6, F8, F2 Instruction Status Table 1: Instruction Issue Execute Write Result Reservation Stations Table 2: ST1 ST2 ST3 Qi Busy Addr Store Status Name Busy Op Vj Vk Qj Qk Add1 Add2 Add3 Mult1 Mult2 Register Status Table 3: F0 F2 F4 F6 F8 F10... F30 Chapter 4 page 39

40 Blank Forms LD1 LD2 LD3 LD4 LD5 LD6 Busy Addr Load Status LD F6, 34(R2) LD F2, 45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10, F0, F6 ADDD F6, F8, F2 Instruction Status Table 1: Instruction Issue Execute Write Result Reservation Stations Table 2: ST1 ST2 ST3 Qi Busy Addr Store Status Name Busy Op Vj Vk Qj Qk Add1 Add2 Add3 Mult1 Mult2 Register Status Table 3: F0 F2 F4 F6 F8 F10... F30 Chapter 4 page 40

41 Blank Forms LD1 LD2 LD3 LD4 LD5 LD6 Busy Addr Load Status LD F6, 34(R2) LD F2, 45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10, F0, F6 ADDD F6, F8, F2 Instruction Status Table 1: Instruction Issue Execute Write Result Reservation Stations Table 2: ST1 ST2 ST3 Qi Busy Addr Store Status Name Busy Op Vj Vk Qj Qk Add1 Add2 Add3 Mult1 Mult2 Register Status Table 3: F0 F2 F4 F6 F8 F10... F30 Chapter 4 page 41

42 Blank Forms LD1 LD2 LD3 LD4 LD5 LD6 Busy Addr Load Status LD F6, 34(R2) LD F2, 45(R3) MULTD F0, F2, F4 SUBD F8, F6, F2 DIVD F10, F0, F6 ADDD F6, F8, F2 Instruction Status Table 1: Instruction Issue Execute Write Result Reservation Stations Table 2: ST1 ST2 ST3 Qi Busy Addr Store Status Name Busy Op Vj Vk Qj Qk Add1 Add2 Add3 Mult1 Mult2 Register Status Table 3: F0 F2 F4 F6 F8 F10... F30 Chapter 4 page 42

43 Key Components of Tamosulo's Algorithm Benefit is Total Data Driven Dynamic memory disambiguation allowing loads and stores to be issued if addresses are different Hence: stall loads if there is a pending store with a matching address wait until the store is done then issue the load likewise, stall stores if there is a pending load from a previous instruction -- note that memory address conflict cannot be resolved by register renaming Register Renaming & Buffering of Sources source operand buffering resolves WAR hazards virtual registers and tagging resolve RAW and WAW Consider the following example: How are the hazards resolved? ADDD F1,F1,F3 followed by ADD F1,F1,F4 Chapter 4 page 43