Project Olympus 3U PCIe Expansion Server. 1/23/2019 Mark D. Chubb

Transcription

1 Project Olympus 3U PCIe Expansion Server 1/23/219 Mark D. Chubb 1

2 Project Olympus HW Building Blocks Motherboards: Flexible expansion Power Supply: 1KW three-phase Universal PMDU: AC power cord adaption Rack: 19 EIA-31 standard Rack Manager: Restful API or Redfish via Ethernet 2

3 3U PCIe Expansion Server Project Olympus Rack Fully Compatible Supports 3 x Load Balanced 1kW Project Olympus PSUs Power Distribution Board (PDB) Supports Project Olympus motherboards PSUs Blind mate into Rack PMDU 6 x 6mm Dual Rotor Fans (room for 8 Fans) 2.5 /3.5 SSD or HDD Supports Slide Rails for smoother travel 5 x FHFL, Double-wide 3W x16 PCIe Cards + 1 x FHHL Single-wide 75W x16 PCIe Card OR 6 x FHFL, Double-wide 3W x16 PCIe Cards + 1 x FHHL Single-wide 75W x16 PCIe Card 12 x FHFL, Single-wide 75W x16 PCIe Cards + 1 x FHHL Single-wide 75W x16 PCIe Card AMD SKU Intel SKU 3

4 ZT Solution: 3U PCIe Expansion Server (AMD EPYC Based) Feature Qty Description Chassis 1 3U, 19 EIA31-D Compliant supporting Project Olympus PMDU connections Motherboard 1 Project Olympus 2-Slot AMD EPYC Motherboard (OCP leveraged) Processor 2 AMD EPYC 7551 (Naples), 18W, 32C, 2GHz, PS7551BDVIHAFV Memory 16 (32max) 32GB DDR4, DR (1DPC), 2667 R-s; Total System Memory: 512GB (Limited to 24MT/s based on Signal Integrity) PCIe Riser 3 & Slot Active PCIe x16 Riser Cards with 48-lane PCIe Switch PCIe Riser Slot Passive PCIe x32 Riser Card GPU Card 4 AMD MI25 FHFL, Double-Wide, 3W PCIe x16 Card (5max) Network Card 1 1G Single Port SFP+ PCIe2. x8 5GT/s HDD/SSD 5 M.2 96GB NVMe SSDs, PCIe x4 11mm (8max) M.2 Riser 1 2-Slot M.2 Riser (in PCIe Slot 1) (2max) Security 1 TPM2. SPI Module System Fans 6 6mmx56mm Dual Rotor Fans Power Supply 3 P21 12W 3-Phase, non-les PSUs Power Distribution 1 PDB and Cable Harnesses to support 12V Power to MB, Risers, GPUs, and System Fans 4

5 3U PCIe Expansion System Options ZT Systems XPO2 3U PCIe Expansion System: Featuring AMD EPYC Platform with 5 x AMD MI25 GPU Cards ZT Systems XPO2 3U PCIe Expansion System: Featuring Intel Purley Platform with 12 x Nvidia P4 GPU Cards or 6 x Nvidia V1 GPU Cards 5

6 48 Mb/s System Block Diagram CHA GMI2 GMI GMI CHA CHB GMI3 GMI GMI GMI GMI1 GMI2 CHB CHC GMI1 GMI GMI3 CHC CHE CHD BIOS Flash 32 MB 33MHz SPI CHE CHD CHF CPU1 BIOS Flash 32 MB CPU CHF PCIe x32 SLOT #4 CHG CHH P[15:] P2[15:] P1[15:] P3[7:4] P3[3:] P3[15:8] USB1(3) TPM Mb/s Internal USB 2. Header Internal USB 2. USB1(3) LPC SMBUS P1[12] P1[11:8] P1[7:4] P1[] P1[1] P1[2] P1[3] P3[7:] USB() USB(1) P[7:] P[15:8] P2[15:] P3[11:8] P3[15:12] CHG CHH 6 Gb/s 6 Gb/s 6 Gb/s SATA x1 SATA1 x1 SATA2 x1 PCIe x16 SLOT #2 6 Gb/s SATA3 x1 PCIe x16 SLOT #1 M.2 Module #3 M.2 Module #4 Oculink x8 48 Mb/s GL85G USB2. HUB Gen3 x8 BMC5781 1G Controller 1 Gb/s 1 Gb/s Front Panel SFP+ Front Panel SFP+ PCIe x16 SLOT #5 BMC Flash #1 32 MB SPI USB2A USB2B 48 Mb/s 5 Gb/s Front Panel USB 3. Port PEX8747 Port 8 Port 16 PCIe x16 SLOT #2 BMC Flash #2 32 MB DDR4 512 MB DDR4 BMC LPC SMBUS PCIe Gen2 x1 5 Gb/s Gen3 x8 Gen3 x8 Front Panel USB 3. PCIE x8 SLOT #1 PCIE x8 SLOT #2 PCIe x16 SLOT #1 Debug Connector VGA DE-15 UART VGA Gen3 x4 Gen3 x4 M.2 Module #1 M.2 Module #2 PCIe x16 SLOT #3 RJ45 BCM GBe PHY RGMII NCSI NCSI Connector Gen3 x4 or 4 SATA 6 Gb/s Mini SAS HD x4 #1 Port PEX8747 Gen3 x4 Mini SAS HD x4 #2 Port 8 Port 16 PCIe x16 SLOT #2 PCIe x16 SLOT #1 6

7 2-Slot Active PCIe x16 Riser Card Layout PCIe Connectors (not shown) are on Topside 2 x PCIe x16 slots supporting (Slots #1-2, Bottom to Top): 2 x FHFL Double-Wide PCIe cards Requires PWRBRK_N (pin B3) signal for High Power PCIe Card throttling Broadcom 48-Lane PCIe Gen3 Switch (PEX8747) Switch Bifurcated into 2 x16 Ports (1 x16 Port to each x16 Slot) Switch is located on Backside as required by Chassis Thermals Use of all 48-lanes and requires 15.5W Heatsink is required Differential Clock Buffer on board for additional Clock Fanout I2C MUX used to avoid Address contention (see I2C block diagram) 8 Layer Stackup, Mid-Loss Material (.62 ) 7

8 2-Slot Passive PCIe x32 Riser Card Layout 1 x PCIe x16 slot supporting (Slot #1, Bottom): 1 x FHHL Single-Wide PCIe card Designated for the Network Card 1 x PCIe x16 slot supporting (Slot #2, Top): 1 x FHFL Single-Wide PCIe Card OR 1 x FHFL Double-Wide PCIe Card All x16 PCIe lanes sourced from PCIe Riser Slot #4 (No Switch Required) Requires PWRBRK_N (pin B3) signal for High Power PCIe Card throttling I2C MUX used to avoid Address contention (see I2C block diagram) 8 Layer Stackup, Mid-Loss Material (.62 ) 8

9 PCIe I2C Block Diagram P1V8_PCH_STBY _STBY 33 I2C I2C_ 2.2K 2.2K PCA9617 PCA9617 I2C_2 Riser 1, PCIe x8 BMC AST252 Bus Naming: 1-based pin -based int PCA9548A xe I2C Notes: M.2 Slots labeled #1-4 in MB Silkscreen. M.2 I2C Addresses (VPD: xa6; Mgmt: x3a) * based TCA9548A xe2 * based I2C_2 I2C_3 I2C_4 I2C_5 I2C_6 I2C_7 I2C_ I2C_2 I2C_3 I2C_[7:4] M.2 S3 M.2 S2 M.2 S1 M.2 S _STBY _STBY _STBY NA P1V8_P NA P1V8_P PCA9617 PCA9617 _STBY Riser 3 _STBY Riser4(x32) _STBY Riser 5 PCA9548A xe4 I2C I2C_ I2C_2 I2C_3 I2C_4 I2C_5 I2C_6 I2C_7 PCA9548A xe4 I2C I2C_ I2C I2C_2 I2C_3 I2C_4 I2C_5 I2C_6 I2C_7 PCA9548A xe4 I2C I2C_ I2C_2 I2C_3 I2C_4 I2C_5 I2C_6 I2C_7 _STBY _STBY _STBY _STBY Tem p sensor xf8(topside) 22 _STBY I2C_ Voltage Monitor x9 _STBY _STBY _STBY _STBY _STBY Tem p sensor xf8(topside) 22 Temp sensor (EMC1413) xf8 Tem p sensor x98(backside) Voltage Monitor x9 Tem p sensor x98(backside) Riser 2, PCIe x8 PCA9617 P1V8 FRU xac Riser 1, PCIe x8 _STBY Riser 2, PCIe x8 FRU(24C64) xac PCA9617 P1V8 FRU xac _SL OT2 _SL OT1 Slot2,PCIe x16 Slot1,PCIe x16 Slot2,PCIe x16 Slot1,PCIe x16 Slot2,PCIe x16 Slot1,PCIe x16 Physical Slot # s Slot 3.4 Slot 3.2 PEX8747 x7 CLK Buffer xd8 Physical Slot # s Slot 4.2 PEX8747 x7 Slot 4.1 Physical Slot # s Slot 5.4 Slot 5.2 CLK Buffer xd8 9

10 PCIe Clock Block Diagram 1

11 Power Distribution Board Layout Power Harness for 2 x AMD MI25 GPUs PDB supports single 12V Power Domain from 3 x Project Olympus PSUs Uses Passive Droop Sharing No V_Sense or I_Share signals used Power Delivery Requirements Hot Swap Controller for Power Monitoring and Isolation 12V Current Sense Ckts used for each branch Supports Power to entire System (MB, Risers, PCIe Cards) Fan Power also comes from PDB (none from MB) Provides passthrough of PMDU signals between MB and RM 6 Layer Stackup (.93 ), 2oz copper planes Different power harnesses can be used for different Cards 11

12 P12V_PSU_SW P12V_STBY PDB Power Distribution Block Diagram Rev 1.2 1U Fan Power Conn Not Used 1U Fan Power & HDD PWR Conn Power Distribution Board MB PWR Conn PSU1 xb 2x12 P12V_PSU 2x12 HSC/ SW PSU2 xb 2x12 Hot Swap Controller (ADM1172) Sense 2x4 Fan Power (6 x 6mm) Riser 3 (2-Slot) Sense 2x6 PSU3 xb 2x12 Gen5 Riser _CB VR Two 2x4 GPU Note: Motherboard has its own Hot Swap Controller & Switch Ckt. Hot Swap Controller (ADM1278) _STBY VR SW Sense _RISER4 Two 2x4 2x2 GPU Riser 4 (2-Slot x32) Riser 5 (2-Slot) Sense 2x6 Two 2x4 GPU Two 2x4 GPU 12

13 PSU LED Cable Sense Sense Sense Sense 2U Conn Jumper 1U Conn PDB Misc Signal Block Diagram MB PDB Rev 1.4 BMC MM2 SMB_BMC_MM2_SCL/SDA I2C_PSU1_SCL/SDA PSU1 PMBUS (xb) PMDU GPIOM5 PSU2_ALERT_N GPIOF4 GPIOJ GPIOF3 GPIX[5:] MM3 MM8 PSU1_ALERT_N CPLD PL5A PSU1_BLADE_EN_N PSU1_THROTTLE_N BMC_NODE_ID_[5:] Throttle Logic SMB_BMC_MM3_SCL/SDA SMB_BMC_MM8_SCL/SDA 1K SCL/SDA RESET# SMBUS MUX PCA9546A (xe) SC/SD SC1/SD1 SC2/SD2 SC3/SD3 Not Used Not Used OD PSU1_ALERT_N PSU1_PS_ON_N PSU1_BLADE_EN_N PSU1_THROTTLE BMC_NODE_ID_[5:] PSU1_G_LED PSU1_Y_LED PSU1_PRESENT_N I2C_PSU2_SCL/SDA PSU_ALERT# PS_ON# PSU_LED PSU_LED1 PSU2 PMBUS (xb) BLADE_EN# BLADE_THROTTLE NODE_ID[5:] BLADE_PRESENT# _STBY MB HSC SN14239 (x8) CPLD PL3C Note: Cabled LEDs mount directly to front panel of Chassis. PSU1_G_LED PSU1_Y_LED PSU2_G_LED PSU2_Y_LED PSU3_G_LED PSU3_Y_LED PSU2_PS_ON_N FRU xa6 PDB HSC ADM1278 (x26) Temp Sensor xf8 SCL/SDA HSC_P12V_CSOUT I2C I/O Expander PCA9555 (x4) IO IO1 IO3 IO12 IO2 IO13 IO1 IO11 IO4 IO14 IO5 IO15 IO6 IO16 IO7 IO17 Comparator Circuit PSU2_PS_ON_BUF_N PSU3_PS_ON_BUF_N RST_I2C_MUX_N HSC_P12V_FAULT_N EN_HSC_IN PDB_CURRENT_ALERT_N PSU2_BLADE_EN_R_N PSU3_BLADE_EN_R_N RISER3_OC_CLEAR RISER3_OC_ALERT_N RISER4_OC_CLEAR RISER4_OC_ALERT_N RISER5_OC_CLEAR RISER5_OC_ALERT_N FAN_OC_CLEAR FAN_OC_ALERT_N OD PSU2_ALERT_N PSU2_PS_ON_N PSU2_BLADE_EN_N NC NC PSU2_G_LED PSU2_Y_LED PSU2_PRESENT_N I2C_PSU3_SCL/SDA OD PSU3_ALERT_N PSU3_PS_ON_N PSU3_BLADE_EN_N NC NC PSU3_G_LED PSU3_Y_LED PSU3_PRESENT_N PSU_ALERT# PS_ON# PSU_LED PSU_LED1 PSU3 PMBUS (xb) PSU_ALERT# PS_ON# PSU_LED PSU_LED1 BLADE_EN# BLADE_THROTTLE NODE_ID[5:] BLADE_PRESENT# BLADE_EN# BLADE_THROTTLE NODE_ID[5:] BLADE_PRESENT# 13