Enabling Arm DynamIQ support. Dan Handley (Arm) Ionela Voinescu (Arm) Vincent Guittot (Linaro)

Transcription

1 Enabling Arm DynamIQ support Dan Handley (Arm) Ionela Voinescu (Arm) Vincent Guittot (Linaro)

2 Agenda DynamIQ introduction DynamIQ and Arm Trusted Firmware OS Power Management with DynamIQ L3 partial power-down support ENGINEERS AND DEVICES WORKING TOGETHER

3 DynamIQ key features From 1. A new single-cluster design 2. Intelligent compute capabilities 3. Interfaces for closely coupled accelerators 4. Built-in power-saving features 5. DynamIQ big.little 6. Advanced RAS and safety features ENGINEERS AND DEVICES WORKING TOGETHER

4 DynamIQ key features From 1. A new single-cluster design 2. Intelligent compute capabilities 3. Interfaces for closely coupled accelerators 4. Built-in power-saving features 5. DynamIQ big.little 6. Advanced RAS and safety features ENGINEERS AND DEVICES WORKING TOGETHER

5 DynamIQ Shared Unit (DSU) TRM: Armv8.2+ Cortex-A CPU support e.g. Cortex-A55, Cortex-A75 2 different CPU types in same cluster Maximum 8 Per-CPU L1+L2 caches and shared L3 Per-CPU DVFS control Partial L3 cache power down Hardware assisted power management Simplifies power up/down software

6 Agenda DynamIQ introduction DynamIQ and Arm Trusted Firmware OS Power Management with DynamIQ L3 partial power-down support ENGINEERS AND DEVICES WORKING TOGETHER

7 DynamIQ Shared Unit (DSU) and Arm TF DSU enables simpler, faster and more robust software during power up/down Simplified micro-architectural programming sequence Automatic enabling and disabling of coherency with the interconnect Automatic and faster cache flushing at all levels without software intervention Reduced power controller communication via P-channel interface TF enables more performant PSCI operations via HW_ASSISTED_COHERENCY option CPU idle, hotplug, secondary CPU boot Will still work without HW_ASSISTED_COHERENCY but won t get the benefits Allows more aggressive OSPM tuning Warning: Some HW operations will be invisible to SW and may give misleading statistics

8 CPU idle to power down (Armv8.0 CPUs) OS calls SMC CPU_SUSPEND Power Down Validate CPU_SUSPEND arguments Acquire locks for non-cpu levels PSCI state coordination CPU-specific power down handling Disable data caches Flush data cache(s) Disable intra-cluster coherency (!SMP_BIT) Stack maintenance Platform suspend operations Release locks for non-cpu levels Wait For Interrupt (WFI) Reset Power Up Minimal SCTLR initialization Platform reset handling CPU-specific reset handling Errata handling Enable intra-cluster coherency (SMP_BIT) CPU architectural register initialization Enable MMU Acquire locks for non-cpu levels Platform suspend-finish operations Stack maintenance Enable data caches Restore OS context PSCI bookkeeping Release locks for non-cpu levels ERET to OS

9 CPU idle to power down (Armv8.2 CPUs) OS calls SMC CPU_SUSPEND Power Down Validate CPU_SUSPEND arguments Acquire locks for non-cpu levels PSCI state coordination CPU-specific power down handling Request CPU power down (CORE_PWRDN_EN) Platform suspend operations Release locks for non-cpu levels Wait For Interrupt (WFI) Reset Minimal SCTLR initialization Platform reset handling CPU-specific reset handling Errata handling (none yet) CPU architectural register initialization Enable MMU and data caches Acquire locks for non-cpu levels Platform suspend-finish operations Restore OS context PSCI bookkeeping Release locks for non-cpu levels ERET to OS Power Up

10 CPU idle to power down (Armv8.2 CPUs) OS calls SMC CPU_SUSPEND Validate CPU_SUSPEND arguments Acquire locks for non-cpu levels PSCI state coordination CPU-specific power down handling Request CPU power down (CORE_PWRDN_EN) Platform suspend operations Release locks for non-cpu levels Wait For Interrupt (WFI) D$ remains enabled throughout Power Down D$ enabled much earlier Reset Minimal SCTLR initialization Platform reset handling CPU-specific reset handling Errata handling (none yet) CPU architectural register initialization Enable MMU and data caches Acquire locks for non-cpu levels Platform suspend-finish operations Restore OS context PSCI bookkeeping Release locks for non-cpu levels ERET to OS Power Up

11 CPU idle to power down (Armv8.2 CPUs) OS calls SMC CPU_SUSPEND Power Down Validate CPU_SUSPEND arguments Acquire locks for non-cpu levels PSCI state coordination CPU-specific power down handling Request CPU power down (CORE_PWRDN_EN) Platform suspend operations Release locks for non-cpu levels Wait For Interrupt (WFI) No need for explicit cache flushes or stack maintenance Reset Minimal SCTLR initialization Platform reset handling CPU-specific reset handling Errata handling (none yet) CPU architectural register initialization Enable MMU and data caches Acquire locks for non-cpu levels Platform suspend-finish operations Restore OS context PSCI bookkeeping Release locks for non-cpu levels ERET to OS Power Up

12 CPU idle to power down (Armv8.2 CPUs) OS calls SMC CPU_SUSPEND Power Down Validate CPU_SUSPEND arguments Acquire locks for non-cpu levels PSCI state coordination CPU-specific power down handling Much more efficient spin Request locks instead CPU power of bakery down locks (CORE_PWRDN_EN) (using v8.1 CAS instruction) Platform suspend operations Release locks for non-cpu levels Wait For Interrupt (WFI) Reset Minimal SCTLR initialization Platform reset handling CPU-specific reset handling Errata handling (none yet) CPU architectural register initialization Enable MMU and data caches Acquire locks for non-cpu levels Platform suspend-finish operations Restore OS context PSCI bookkeeping Release locks for non-cpu levels ERET to OS Power Up

13 CPU idle to power down (Armv8.2 CPUs) OS calls SMC CPU_SUSPEND Power Down Validate CPU_SUSPEND arguments Acquire locks for non-cpu levels PSCI state coordination CPU-specific power down handling Request CPU power down (CORE_PWRDN_EN) Platform suspend operations Release locks for non-cpu levels Wait For Interrupt (WFI) (Potentially) reduced power controller communication Reset Minimal SCTLR initialization Platform reset handling CPU-specific reset handling Errata handling (none yet) CPU architectural register initialization Enable MMU and data caches Acquire locks for non-cpu levels Platform suspend-finish operations Restore OS context PSCI bookkeeping Release locks for non-cpu levels ERET to OS Power Up No need for explicit interconnect programming for masters to enter/exit coherency

14 Future TF enhancements Use per-thread cluster power voting register CLUSTERPWRDN_EL1 Automatic cluster power down or memory retention if the power controller hardware and firmware support it Remove cluster level locks or at least reduce the time they are held Analyze performance on DynamIQ hardware platforms ENGINEERS AND DEVICES WORKING TOGETHER

15 Agenda DynamIQ introduction DynamIQ and Arm Trusted Firmware OS Power Management with DynamIQ L3 partial power-down support ENGINEERS AND DEVICES WORKING TOGETHER

16 OS Power Management with DynamIQ Finer grained power capabilities Already handled by PM frameworks Per-core Frequency/Voltage domain DSU Frequency/Voltage domain

17 Scheduler domains Current big.little system Example of 4 big cores + 4 LITTLE cores: Energy model layout matches scheduler domain

18 Scheduler domains DynamIQ changes domains boundaries Not necessarily congruent Physical / Voltage / Frequency / Architecture Change the scheduler topology And energy model layout Example of 4 big cores + 4 LITTLE cores:

19 Phantom domains Add intermediate domain Voltage/Frequency boundary Example of 4 big cores + 4 LITTLE cores: Per core DVFS

20 Phantom domains Example of 4 big cores + 4 LITTLE cores: One frequency domain for big cores and one for LITTLE cores Frequency domain close to current big.little system Enable similar scheduler topology

21 OSPM next steps Shared frequency domains Shared voltage domains Impact on energy model Impact on compute capacity Getting notified of power domain OPP change Multiple DynamIQ clusters ENGINEERS AND DEVICES WORKING TOGETHER Reference:

22 Agenda DynamIQ introduction DynamIQ and Arm Trusted Firmware OS Power Management with DynamIQ L3 partial power-down support ENGINEERS AND DEVICES WORKING TOGETHER

23 L3 partial power-down Arm DynamIQ Shared Unit (DSU) L3 cache Implementation specific number of portions controlled through a power control register Counters for cache misses and cache hits to help drive decisions ENGINEERS AND DEVICES WORKING TOGETHER Support in software DevFreq driver Control of active portions based on: Cache hit/miss rates Computed power benefit Bias for performance Out of tree reference implementation:

24 L3 partial power-down: architecture Linux Kernel DSU register interface DSU L3 cache DevFreq governor Target portions hit counter miss counter Timer 10ms DevFreq device Set target portions control register Update DevFreq

25 L3 partial power-down: algorithm Upsize: Weigh additional cost in energy of enabling another portion against potential savings by decreasing dynamic cost of accessing DRAM. Condition for upsize: MBW > (1.0 Tu) * CB Compare energy consumption Bias for performance L3 cache static MBW miss bandwidth: MiB/sec CB cost bandwidth: MiB/sec CB = L / ED L static leakage of single portion: uj/sec ED dynamic energy of DRAM: uj/mib Tu upsizing threshold: fraction 0.00 to 1.00 Bias for performance DRAM dynamic energy

26 L3 partial power-down: algorithm - 1 Downsize: From an energy trade-off perspective, to justify a portion to be powered on, requires a hit bandwidth that pays for its leakage. If that requirement is not met, it can be powered-off. Compare energy consumption Bias for performance Condition for downsize: HBW < (N Td) * CB DRAM dynamic energy HBW hit bandwidth: MiB/sec N current number of portions enabled CB cost bandwidth: MiB/sec CB = L / ED L static leakage of single portion: uj/sec ED dynamic energy of DRAM: uj/mib Td downsize threshold: fraction 0.00 to 1.00 Bias for performance L3 cache static

27 L3 partial power-down: behaviour Example: 2MB L3 cache Memcpy workload with buffer size of 4MB

28 L3 partial power-down: behaviour - 1 Expected behaviour: CPU intensive workloads should not have an effect on the number of active portions I/O intensive loads should raise portions when the cache is well used

29 L3 partial power-down Limitations of current reference implementation Portion is the smallest single unit of the cache that can be powered up/down Only support for a single DynamIQ Shared Unit Not suitable for use with the simple on-demand governor L3 partial power-down in Arm Trusted Firmware? ENGINEERS AND DEVICES WORKING TOGETHER Reference:

30 Thank You #SFO17 BUD17 keynotes and videos on: connect.linaro.org For further information: