Evaluation of AMD EPYC

Transcription

1 Evaluation of AMD Chris Hollowell HEPiX Fall 2018, PIC Spain

2 What is? is a new line of x86_64 server CPUs from AMD based on their Zen microarchitecture Same microarchitecture used in their Ryzen desktop processors Released June 2017 First new high performance series of server CPUs offered by AMD since 2012 Last were Piledriver-based Opterons Steamroller Opteron products cancelled AMD had focused on low power server CPUs instead x86_64 Jaguar APUs ARM-based Opteron A CPUs Many vendors are now offering -based servers, including Dell, HP and Supermicro 2

3 How Does Differ From Skylake-SP? Intel s Skylake-SP Xeon x86_64 server CPU line also released in 2017 Both Skylake-SP and CPU dies manufactured using 14 nm process Skylake-SP introduced AVX512 vector instruction support in Xeon AVX512 not available in HS06 official GCC compilation options exclude autovectorization Stock SL6/7 GCC doesn t support AVX512 Support added in GCC 4.9+ Not heavily used (yet) in HEP/NP offline computing Both have models supporting memory Skylake-SP 6 memory channels per processor 3 TB (2-socket system, extended memory models) 8 memory channels per processor 4 TB (2-socket system) 3

4 How Does Differ From Skylake (Cont)? Some Skylake-SP processors include built in Omnipath networking, or FPGA coprocessors Not available in Both Skylake-SP and have SMT (HT) support 2 logical cores per physical core (absent in some Xeon Bronze models) Maximum core count (per socket) Skylake-SP 28 physical / 56 logical (Xeon Platinum 8180M) 32 physical / 64 logical ( 7601) Maximum socket count Skylake-SP 8 (Xeon Platinum) 2 Processor Inteconnect Skylake-SP UltraPath Interconnect (UPI) EYPC Infinity Fabric (IF) PCIe lanes (2-socket system) Skylake-SP (some used by SoC functionality) Same number available in single socket configuration 4

5 : MCM/SoC Design utilizes an SoC design Many functions normally found in motherboard chipset on the CPU SATA controllers USB controllers etc. Each processor consists of four CPU dies, interconnected via Infinity Fabric Multi-Chip Module (MCM) architecture CPU Complexes (CCX) Each CCX attached to its own memory 2 memory channels per CCX All Skylake-SP cores are on a single die AMD claims MCM results in a cost reduction by improving yields Believed to scale better than monolithic die approach as core counts continue to increase Drawback: higher memory latency for non-numa-aware applications 5

6 : MCM/SoC Design (Cont.) # lscpu Architecture: CPU op-mode(s): Byte Order: CPU(s): On-line CPU(s) list: Thread(s) per core: Core(s) per socket: Socket(s): NUMA node(s): Vendor ID: CPU family: Model: Model name: Processor Stepping: CPU MHz: CPU max MHz: CPU min MHz: BogoMIPS: Virtualization: L1d cache: L1i cache: L2 cache: L3 cache: NUMA node0 CPU(s): NUMA node1 CPU(s): NUMA node2 CPU(s): NUMA node3 CPU(s): NUMA node4 CPU(s): NUMA node5 CPU(s): NUMA node6 CPU(s): NUMA node7 CPU(s): x86_64 32-bit, 64-bit Little Endian AuthenticAMD 23 1 AMD Core AMD-V 32K 64K 512K 8192K 0-3, , , , , , , ,60-63 # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 72 On-line CPU(s) list: 0-71 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold GHz Stepping: 4 CPU MHz: BogoMIPS: Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38, 40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39, 41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71 vs Skylake-SP (SNC Disabled) NUMA Configuration 6

7 Socket LGA 3647 & SP3 Skylake SP Socket LGA 3647 Socket SP3 Both CPUs/sockets are quite large Visible quadrants in the SP3 socket for the four CPU complexes in the processor 7

8 Skylake and Model Lineup Comparison Model Base Frequency Cores SMT TDP Memory Retail Xeon Bronze GHz (no turbo) 6 No 85W 2133 MHz DDR4 $213 Xeon Silver GHz 8 85W 2400 MHz DDR4 $501 Xeon Gold GHz 10 85W $1,221 Xeon Gold GHz W $1,900 Xeon Gold GHz W $2,460 Xeon Gold GHz W $3,072 Xeon Gold GHz W $3,358 Xeon Platinum GHz W $7,405 Xeon Platinum 8180M 2.5 GHz W $13, GHz 8 120W 2400 MHz DDR4 $ GHz W $1,110 Uniprocessor (P) - $ GHz W $1,850 Uniprocessor (P) - $1, GHz W $2, GHz W $3, GHz W $4,200 8

9 vs Skylake-SP: HEP/NP Performance Benchmarks HEPSPEC06 all_cpp subset of SPEC-CPU2006 run in parallel CERN Cloud Benchmark Suite Various benchmarks, run in parallel DB12 Whetstone ATLAS KV Unless noted, memory configured to utilize all 8 channels per CPU on, and 6 channels per CPU for Skylake-SP, with at least 2 GB RAM/logical core ~11% HS06 performance degradation seen for 7441 when only populating half of the memory channels All Noted dual rank (DR) DIMMs downclocked to 2400 MHz for All run under SL/CentOS/RHEL 7 SMT/Hyperthreading enabled, unless otherwise indicated Systems are dual CPU, unless noted 9

10 HEPSPEC06: SMT Off vs On 25%+ HS06 performance improvement with SMT ( hyperthreading ) enabled P P SMT SMT HS SMT SMT SMT P@2.0 GHz [Uniprocessor - 24 threads] 7401P@2.0 GHz [Uniprocessor - 48 threads] 7351@2.4 GHz [32 threads] 7351@2.4 GHz [64 threads] 7451@2.3 Ghz [48 threads] DDR @2.3 Ghz [96 threads] DDR @2.0 GHz [64 threads] 7551@2.0 GHz [128 threads] 7601@2.2 GHz [64 threads] DDR @2.2 GHz [128 threads] DDR CPU 10

11 vs Skylake-SP: HEPSPEC P Xeon Gold Xeon Gold Xeon Gold Xeon Gold 6150 HS06 Xeon Gold Xeon Platinum Xeon Gold GHz [40 threads] + Xeon Gold 6130@2.1 GHz [64 threads] Xeon Gold 6136@3.0 GHz [48 threads] Xeon Gold 6148@2.4 GHz [80 threads] Xeon Gold 6150@2.7 GHz [72 threads] Xeon Platinum 8170@2.1 GHz [104 threads] * 7401P@2.0 GHz [Uniprocessor - 48 threads] 7351@2.4 GHz [64 threads] 7451@2.3 Ghz [96 threads] DDR @2.0 GHz [128 threads] 7601@2.2 GHz [128 threads] DDR = System using only 3 memory channels per CPU * = Value reported by CERN 0 CPU 11

12 vs Skylake: HEPSPEC06 (Cont.) Larger values are better Similar maximum HS06 (~1,275) performance for the models tested Data for highest level (7601), but not highest model Skylake-SP (8180M) Can assume Xeon Skylake 8180M would perform better than the 8170 value listed Same number of cores/threads as 8170, but higher clock speed 2.5 GHz vs 2.1 GHz Mid-range model HS06 performance also similar ~700 HS06 - ~1100 HS06 TDP somewhat higher for CPUs vs Xeon Gold, in general 165 W max Xeon Gold, vs 180 W max Can likely expect to use a bit more power as a result 12

13 vs Skylake-SP: CERN Cloud Benchmarks 1400 Dirac HS06 Est Xeon Gold Xeon Gold GHz [40 threads] + Xeon Gold 6150@2.7 GHz [72 threads] 7351@2.4 GHz [64 threads] 7551@2.0 GHz [128 threads] = System using only 3 memory channels per CPU 7551 Xeon Gold Xeon Gold Events/Sec Xeon Gold Xeon Gold Xeon Gold BWIPS DB12 (aggregate) Whetstone (aggregate) ATLAS KV (aggregate) 13

14 /Skylake: CERN Cloud Benchmarks (Cont.) Results for a limited number of CPUs: only possible to run the full suite (including KV) on systems with CVMFS client installed/setup By default the CERN cloud benchmarks run one instance of a benchmark per logical core in parallel However, only reports performance per logical core Interested in aggregate system performance, not performance/logical core For DB12, and Whetstone, simply multiplied result by number of logical cores For KV, average seconds/event per logical core is reported Took the inversion, and multiplied by the number of logical cores to obtain total events/sec Larger graphed values are better DB12 and Whestone results fairly in line with HS06 Expected somewhat better KV performance for the Xeon Gold

15 vs Skylake-SP: CPU HS06/Dollar Only retail CPU cost accounted for in calculations Does not represent reality given memory and base server pricing P Xeon Platinum 8170 Xeon Gold Xeon Gold Xeon Gold Xeon Gold Xeon Gold 5115 HS06/$ Xeon Gold GHz [40 threads] + Xeon Gold 6130@2.1 GHz [64 threads] Xeon Gold 6136@3.0 GHz [48 threads] Xeon Gold 6148@2.4 GHz [80 threads] Xeon Gold 6150@2.7 GHz [72 threads] Xeon Platinum 8170@2.1 GHz [104 threads] 7401P@2.0 GHz [Uniprocessor - 48 threads] 7351@2.4 GHz [64 threads] 7451@2.3 Ghz [96 threads] DDR @2.0 GHz [128 threads] 7601@2.2 GHz [128 threads] DDR = System using only 3 memory channels per CPU 0 CPU 15

16 vs Skylake-SP: Estimated 25kHS06 Cost Estimated total cost of 25kHS HS06 Assuming $1,500 irreducible server cost, and retail CPU/memory pricing $1k 258 Xeon Gold GHz [12*16 GB DIMMS, 34 servers] Xeon Gold GHz [12*16 GB DIMMs, 23 servers] Xeon Gold GHz [12*16 GB DIMMs, 24 servers] P Xeon Gold Xeon Gold GHz [6*16 GB DIMMs, 64 servers] + Xeon Gold 6136@3.0 GHz [12*8 GB DIMMs, 32 servers] 257 Xeon Gold Xeon Gold Xeon Gold Xeon Gold Xeon Platinum Xeon Platinum 8170@2.1 GHz [12*32 GB DIMMS, 20 servers] 7401P@2.0 GHz [8*16 GB DIMMs, 51 servers] 7351@2.4 GHz [16*8 GB DIMMs, 32 servers] 7451@2.3 GHz [16*16GB DIMMs, 23 servers] 7551@2.0 GHz [16*16 GB, 22 servers]] 7601@2.2 GHz [16*16 GB, 19 servers] = System using only 3 memory channels per CPU 50 0 CPU 16

17 vs Skylake-SP: Est. 25kHS06 Cost (Cont.) Server counts to achieve 25kHS HS06 (+-2%) Majority of compute node cost in CPU and memory Assuming no excessive local storage space or IOPs requirements Typically the case for HEP/NP Retail CPU costs used in estimate Likely to receive volume or competitive discounts Estimate assumes a server without CPUs and memory costs $1,500 Includes power supply, disk, NIC, etc. Only accounts for cost of servers themselves. Associated costs such as racks, network switches, integration, shipping, etc. not included Server vendors typically increase base prices for servers which support higher performing CPU models with higher TDP (i.e. due to bigger PSUs, etc.) $1,500 server base cost may be lower than reality for systems with higher end CPUs Memory costs: retail Samsung server DIMM pricing DDR MHz, ECC, registered 8 GB DIMM - $ GB DIMM - $ GB DIMM - $380 17

18 vs Skylake-SP: Est. 25kHS06 Cost (Cont.) Enough memory populated per system to provide 2 GB/logical core Fairly standard for HEP/NP Ensured all 6 (Skylake) or 8 () memory channels per CPU utilized for maximum bandwidth, and NUMA performance Often ended up with more RAM than required to satisfy 2 GB/logical core 6 channels makes this particularly difficult for Skylake: installed memory not a power of two Problem compounded by server manufacturers not offering smaller (i.e. 4 GB), less expensive DDR4 DIMMs Estimated cost for 25kHS06 fairly similar between Skylake-SP Xeon Gold and servers: most close to $250k Dual-CPU 7351 systems appear very cost effective, however: est. $189k ~25% less than the Xeon Gold 6148 Required memory/logical core and DIMM channel parity CPU itself is inexpensive compared to other Skylake/ counterparts 7401P uniprocessor system HS06/$ for CPU cost initially looked promising Large number of servers required: irreducible per server cost added up Estimated in the $250k range like many of the other CPUs 18

19 Side-Channel Attacks Jan New class of side-channel information disclosure vulnerabilities in CPU hardware made public Meltdown, Spectre Exploit speculative execution and caching optimizations in CPUs Meltdown Spectre List of similar side-channel attack vectors continues to grow Speculative Store Bypass Vulnerability Foreshadow (L1TF) Microcode updates for Skylake-SP released for all of the above AMD claims is not vulnerable to Meltdown or Foreshadow Due to existing protections in their paging architecture Released microcode updates for Spectre 19

20 The Future: 2 and Cascade Lake 2 - Rome Expected in early nm process Support for DDR MHz DIMMs expected Still 8 channels Max core count per socket increased to 64 (128 threads) AVX512? Cascade Lake Xeon Expected end of nm process 10 nm process expected in Ice Lake in 2020 Max memory speed: DDR DIMMs Still 6 channels Max core count per socket remains at 28 (56 threads) Expected to support VNNI instructions Utilizes AVX512 units Support for Optane 3D Xpoint memory Announced that the Frontera supercomputer at TACC will be Cascade Lake based estimated to provide PFLOPS without GPUs Both CPUs will have Spectre mitigations built into hardware 20

21 Conclusions The MCM architecture is considerably different from Skylake-SP s single die configuration Similar HEP/NP benchmark performance from mid/upper range Skylake-SP Xeon Gold and mid/upper range AMD CPUs Pricing also similar However, dual-cpu 7351-based systems appear to be a sweet spot for applications requiring 2GB/logical core (somewhat typical for HEP/NP software) Competition in the server CPU market will likely help reduce cost and spur innovation 2 (Rome) with its 7nm process and 64 physical cores appears poised to disrupt the existing balance of server CPU market share 21