Instruction-Level Parallelism and its Exploitation: PART 1 ILP concepts (2.1) Basic compiler techniques (2.2) Reducing branch costs with prediction (2.3) Dynamic scheduling (2.4 and 2.5)
Project and Case Studies Project: Lecture Nov 9 Case studies Presentation of pipeline pp case studies on Nov 12 This week: Division into groups (5-6 in each) Selection of case studies: MIPS R10000 Intel Pentium 4 AMD Opteron Sun Rock (UltraSparc) IBM Power 6
Lectures 1. Introduction 2. Instruction-level Parallelism, part 1 3. Instruction-level Parallelism,,part 2 4. Memory Hierarchies 5. Multiprocessors and Thread-Level Parallelism 6. System Aspects and Virtualization 7. Summary and Review
Bottlenecks in Simple Pipelines 4 IF/ID ID/EX EX/MEM MEM/WB P C Data memory Three classes of dependences that limit parallelism: Data hazards (lost cycles due to dependences) Control hazards (lost cycles due to branches) Structural hazards (lost cycles due to lack of resources)
Unlocking Instruction-Level Parallelism: A First Set of Techniques Improve parallel execution in basic pp pipeline to avoid stalls: Static scheduling of instructions (compiler) Dynamic branch prediction (run-time) Exploit parallelism li in programs Add dynamically (run-time) Add for (i=1000; i>0; i=i-1) x[i] = x[i] + 10.0; Add Use many execution units to run things in parallel
Instruction-Level Parallelism Basic Concepts (Ch 2.1) Two instructions must be independent in order to execute in parallel Three classes of dependences that limit parallelism: Data dependences Name dependences d Control dependences Dependences are properties of the program Can lead to hazards which are properties of the pipeline organization
Data Dependences An instruction j is data dependent on instruction i if: instruction i produces a result used by instr. j, or instruction j is data dependent on instruction k and instr. k is data dependent on instr. i Example: [Notation: OP Rx, Ry, Rz Rx <- Ry OP Rz] LD ADDD SD F0,0(R1) F4,F0,F2 0(R1),F4 Easy to detect dependences for registers trickier for memory locations (memory ambiguity/alias problem)
Name Dependences Two instructions use same name (register or memory address) but don t exchange data Anti dependence (WAR if hazard in pipeline) Instruction j writes to a register or memory location that instruction i reads from and instruction i is executed first Output dependence (WAW if hazard in pipeline) Instruction i and instruction j write to the same register or memory location; ordering between instructions must be preserved Name dependences are not fundamental and can sometimes be eliminated through hardware or software techniques (renaming)
Control Dependences Example: if Test1 then { S1 } if Test2 then { S2 } S1 is control dependent on Test1 S2 is control dependent on Test2; but not on Test1 We can t move an instruction that is dependent on a branch before the branch instruction We can t move an instruction that is not control dependent on a branch after the branch instr.
Compiler Techniques to Expose ILP: Example (Ch 2.2) for (i=1000; i>0; i=i-1) x[i] = x[i] + 10.0; Iterations are independent => parallel execution RAW RAW loop: LD F0, 0(R1) ; F0 = array element ADDD F4, F0, F2 ; Add scalar constant SD 0(R1), F4 ; Save result SUBI R1, R1, #8 ; decrement array ptr. BNEZ R1, loop ; reiterate if R1!= 0 NOP ; delayed branch Can we eliminate i all penalties in each iteration? ti
Static Scheduling within each Loop Iteration loop: LD F0, 0(R1) loop: LD F0, 0(R1) stall ADDD F4, F0, F2 stall ADDD F4, F0, F2 stall SUBI R1, R1, #8 stall BNEZ R1, loop SD 0(R1), F4 SD 8(R1), F4 SUBI R1, R1, #8 BNEZ R1, loop NOP Original loop: Statically scheduled loop: Four stall cycles One stall cycle Can we do better by scheduling across iterations?
Loop Unrolling RAW loop: LD F0, 0(R1) RAW ADDD F4, F0, F2 SD 0(R1), F4 ; drop SUBI & BNEZ RAW LD F6, -8(R1) ; adjust displacement ADDD F8, F6, F2 RAW SD -8(R1), F8 ; drop SUBI & BNEZ RAW LD F10, -16(R1) ; adjust displacement ADDD F12, F10, F2 RAW SD -16(R1), F12 ; drop SUBI & BNEZ LD F14, -24(R1) ; adjust displacement RAW ADDD F16, F14, F2 RAW SD -24(R1), F16 SUBI R1, R1, #32 ; alter to 4*8 BNEZ R1, loop NOP Registers must be renamed to avoid WAR hazards A larger chunk of sequential code simplifies scheduling
Statically Scheduled Unrolled Loop loop: LD F0, 0(R1) LD F6, -8(R1) LD F10, -16(R1) LD F14, -24(R1) ADDD F4, F0, F2 ADDD F8, F6, F2 ADDD F12, F10, F2 ADDD F16, F14, F2 SD 0(R1), F4 SD -8(R1), F8 SD -16(R1), F12 SUBI R1, R1, #32 BNEZ R1, loop SD 8(R1), F16 All penalties are eliminated. CPI=1! Important steps: Hoist loads Push stores down Note: the displacement of the store instr. must be changed Effects of loop unrolling: Provides a larger seq. instr. window Simplifies for static and dynamic methods to extract ILP But makes code bigger
Dynamic Methods to Improve ILP Not all potential hazards can be resolved by static scheduling at compile time For static scheduling to work, it is necessary for the compiler to know enough about the processor implementation to predict hazards Therefore, we will also explore dynamic techniques for hazard resolution
General Processor Organization Memory access Fetch instruction Get operands & Issue Integer & Logic Update state Floating point Major bottlenecks Control hazards, memory performance => Fetch bottleneck Data hazards, structural hazards, control hazards => Issue bottleneck
Fetch Bottleneck Control hazards Dynamic branch prediction: Predict outcome of branches and jumps Branch target buffers Issue (and execute) beyond branches Do not update state until prediction verified Memory bottleneck Memory performance improvement (memory hierarchy) Prefetch, multiple fetch
Dynamic Branch Prediction (Ch. 2.3) Branches limit performance because: Branch penalties are high Prevent a lot of ILP from being exploited Solution: Dynamic branch prediction to predict the outcome of conditional branches. Benefits: Reduce time to determine branch condition Reduce time to calculate the branch target address
Branch History Table A simple branch prediction scheme IF ID EX MEM WB PC 1 0 0 1 0 1 0 branches predicted as taken The idea: Use last branch outcome as prediction branch-prediction buffer is indexed by bits from branch- instruction ti PC values If prediction is wrong, then invert prediction Problem: a loop causes two mispredictions in a row
ATwobit Two-bit Prediction Scheme Requires prediction to be wrong twice in order to change prediction => better performance Performance: 0%-18% miss prediction frequency for SPEC92 Integer programs have higher miss frequency than floating point (FP) programs
Correlating Branch Predictors Correlating predictors ((m,n) predictor) Takes into consideration multiple branches and their correlation Performs better than the 2-bit predictor
Issue Bottleneck RAW hazards Dynamic scheduling (out-of-order execution) WAR & WAW hazards Remove name dependencies (register renaming) Structural hazards Dynamic scheduling (out-of-order execution) Memory performance improvement (memory hierarchy, prefetch, non-blocking, load/store buffers) Multiple and pipelined functional units Control hazards Speculative execution Single issue Issue multiple instructions per cycle (superscalar, VLIW)
Dynamic Instruction Scheduling (Ch. 2.4) Key idea: Allow subsequent independent instructions to proceed Instr. gets stuck here DIVD F0,F2,F4 ; takes long time ADDD F10F0F8 F10,F0,F8 ; stalls waiting for F0 SUBD F12,F8,F13 ; Let this instr. bypass the ADDD Enables out-of-order execution => out-of-order completion IF ID EX M WB Two historical schemes used in recent machines: Scoreboard dates back to CDC 6600 in 1963 Tomasulo s s algorithm in IBM 360/91 in 1967
Tomasulo s Algorithm: Hardware Organization (Ch. 2.5) Note: Tomasulo s algorithm is of course applicable also for other types of instructions than floating point.
Basic Ideas Decouple issue from operand fetch Prevents stall due to RAW hazards Register renaming: Translate result register references to instruction (functional unit) references Prevents WAR and WAW hazards Example registers S and T prevent WAW/WAR DIV.D F0,F2,F4 ADD.D F6,F0,F8 S.D F6,0(R1) SUB.D F8,F10,F8 MUL.D F6,F10,F8 DIV.D F0,F2,F4 ADD.D S,F0,F8 S.D S,0(R1) SUB.D T,F10,F8 MUL.D F6,F10,T
Three Stages of Tomasulo s Algorithm 1. Issue get g instruction from FP Op Queue Issue if no structural hazard for a reservation station 2. Execution operate on operands (EX) Execute when both operands are available; if not ready, watch Common Data Bus (CDB) for result 3. Write result finish execution (WB) Write on CDB to all awaiting functional units; mark reservation station available Normal bus: data + destination Common Data Bus: data + source
Tomasulo example, cycle 0 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 Busy Address LD F2 45+ R3 Load1 No MULTDF0 F2 F4 Load2 No SUBD F8 F6 F2 Load3 No DIVD F10 F0 F6 ADDD F6 F8 F2 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No Mult1 No Mult2 No Register result status Clock: 0 FU F0 F2 F4 F6 F8 F10... F30
Tomasulo example, cycle 1 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 Time Busy Address LD F2 45+ R3 Load1 Yes R2+32 MULTDF0 F2 F4 Load2 No SUBD F8 F6 F2 Load3 No DIVD F10 F0 F6 ADDD F6 F8 F2 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No Mult1 No Mult2 No Register result status Clock: 1 FU F0 F2 F4 F6 F8 F10... F30 Load1
Tomasulo example, cycle 2 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 Time Busy Address LD F2 45+ R3 2 1 Load1 Yes R2+32 MULTDF0 F2 F4 Load2 Yes R3+45 SUBD F8 F6 F2 Load3 No DIVD F10 F0 F6 ADDD F6 F8 F2 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No Mult1 No Mult2 No Register result status F0 F2 F4 F6 F8 F10... F30 FU Load2 Load1 Clock: 2
Tomasulo example, cycle 3 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 Time Busy Address LD F2 45+ R3 2 0 Load1 Yes R2+32 MULTDF0 F2 F4 3 1 Load2 Yes R3+45 SUBD F8 F6 F2 Load3 No DIVD F10 F0 F6 ADDD F6 F8 F2 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No Mult1 Yes Mult F4 Load2 Mult2 No Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 Load2 Load1 Clock: 3
Tomasulo example, cycle 4 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 Load1 No MULTDF0 F2 F4 3 0 Load2 Yes R3+45 SUBD F8 F6 F2 4 Load3 No DIVD F10 F0 F6 ADDD F6 F8 F2 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 Yes Sub M(R2+34) Load2 Add2 No Add3 No Mult1 Yes Mult F4 Load2 Mult2 No Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 Load2 - Add1 Clock: 4
Tomasulo example, cycle 5 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 Load2 No SUBD F8 F6 F2 4 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk 2 Add1 Yes Sub M(R2+34)M(R3+45) - Add2 No Add3 No 10 Mult1 Yes Mult M(R3+45) F4 - Mult2 Yes Div F6 Mult1 Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 - Add1 Mult2 Clock: 5
Tomasulo example, cycle 6 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 Load2 No SUBD F8 F6 F2 4 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 6 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk 1 Add1 Yes Sub M(R2+34)M(R3+45) Add2 Yes Add F2 Add1 Add3 No 9 Mult1 Yes Mult M(R3+45) F4 Mult2 Yes Div F6 Mult1 Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 Add2 Add1 Mult2 Clock: 6
Tomasulo example, cycle 7 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 Load2 No SUBD F8 F6 F2 4 7 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 6 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk 0 Add1 Yes Sub M(R2+34)M(R3+45) Add2 Yes Add F2 Add1 Add3 No 8 Mult1 Yes Mult M(R3+45) F4 Mult2 Yes Div F6 Mult1 Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 Add2 Add1 Mult2 Clock: 7
Tomasulo example, cycle 8 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 Load2 No SUBD F8 F6 F2 4 7 8 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 6 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No 2 Add2 Yes Add F6-F2 F2 - Add3 No 7 Mult1 Yes Mult M(R3+45) F4 Mult2 Yes Div F6 Mult1 Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 Add2 - Mult2 Clock: 8
Tomasulo example, cycle 10 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 Load2 No SUBD F8 F6 F2 4 7 8 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 6 10 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No 0 Add2 Yes Add F6-F2 F2 Add3 No 5 Mult1 Yes Mult M(R3+45) F4 Mult2 Yes Div F6 Mult1 Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 Add2 Mult2 Clock: 10
Tomasulo example, cycle 11 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 Load2 No SUBD F8 F6 F2 4 7 8 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 6 10 11 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No 4 Mult1 Yes Mult M(R3+45) F4 Mult2 Yes Div F6 Mult1 Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 - Mult2 Clock: 11
Tomasulo example, cycle 15 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 15 Load2 No SUBD F8 F6 F2 4 7 8 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 6 10 11 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No 0 Mult1 Yes Mult M(R3+45) F4 Mult2 Yes Div F6 Mult1 Register result status F0 F2 F4 F6 F8 F10... F30 FU Mult1 Mult2 Clock: 15
Tomasulo example, cycle 16 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 15 16 Load2 No SUBD F8 F6 F2 4 7 8 Load3 No DIVD F10 F0 F6 5 ADDD F6 F8 F2 6 10 11 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No Mult1 No 40 Mult2 Yes Div F0 F6 - Register result status F0 F2 F4 F6 F8 F10... F30 FU - Mult2 Clock: 15
Tomasulo example, cycle 56 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 15 16 Load2 No SUBD F8 F6 F2 4 7 8 Load3 No DIVD F10 F0 F6 5 56 ADDD F6 F8 F2 6 10 11 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No Mult1 No 0 Mult2 Yes Div F0 F6 Register result status Clock: 56 FU F0 F2 F4 F6 F8 F10... F30 Mult2
Tomasulo example, cycle 57 Instruction status Exec. Write Instruction j k Issue compl. result Load buffers LD F6 34+ R2 1 3 4 Time Busy Address LD F2 45+ R3 2 4 5 Load1 No MULTDF0 F2 F4 3 15 16 Load2 No SUBD F8 F6 F2 4 7 8 Load3 No DIVD F10 F0 F6 5 56 57 ADDD F6 F8 F2 6 10 11 Functional unit status src 1 src 2 RS for j RS for k Time Name Busy Op Vj Vk Qj Qk Add1 No Add2 No Add3 No Mult1 No Mult2 No Register result status F0 F2 F4 F6 F8 F10... F30 FU - Clock: 57
Summary Data, name, and control dependences Hazards and static instruction order prevent ILP exploitation Static methods Let compiler reorder instructions to remove hazards Dynamic methods Modify processor organization to remove hazards and dynamically reorder instructions Fetch bottleneck: Dynamic branch prediction Issue bottleneck: Tomasulo s algorithm Next Speculative execution Multiple issue Register renaming Increased instruction fetch bandwidth