The Why and How of Developing All-Flash Storage Server June 2016 Jungsoo Kim Manager, SK Telecom
Agenda Why we care about All-Flash Storage Transforming to 5G Network Open HW & SW Projects @ SKT Our approaches in developing storage system AF-Media details Future Work for AF-Media NV-Array Development Collaboration 1
Why we care about All-Flash Storage 5G All-Flash Storage Tech. 4K UHD 2
Transforming to 5G Network 10x-1000x speed Massive Connectivity Efficiency & Reliability 5G 10x lower latency Virtualization New infrastructure should be Programmable, Scalable, Flexible, and Cost Effective Software Defined Technologies based on Open Software & Open Hardware 3
Open HW & SW Projects @ SKT Open Software Software-Defined Technologies Open Hardware 4
In the middle of nowhere 5
Our Approaches Building Ecosystem Open Hardware Community Mechanical Reference Designs Design Software Reduce Dev. Cost Improve Tech Skills Test & Verification Tech Support Shorten Dev. Cycle Promotion Effect Simulation Test & Verification 6
How we utilize Open Compute Project Larger Fan Design Reference Shared Power Shared Chassis Shared Backplane Front-Access Design Energy Efficiency & Usability AF-Media Tool-less Design Standardization OCP Mezzanine Mini SAS HD Cable OpenBMC Development & BOM Cost/Cycle NV-Array Collaboration Retimer Card H/W Management Architecture NVMe SSD Development & Verification Efforts Contribution AF-Media : Targeted in July, 2016 NV-Array : Planned in 2017 Ecosystem 7
AF-Media : All-Flash Storage Server High Density High Bandwidth Availability Low Power SATA SSD 16ea in 1U size High Capacity High IO Performance 40GbE Network Support Node Hot-Plug & SSD Hot-Swap Support Under 150W/Node Application AF-Media S16 Media Streaming High Perf. File Server 2RU Node 1 Node 2 CDN Edge Server Proxy Cache Server For Each Server, Network Throughput : ~ 40Gbps Storage Capacity : 32TB/Node (w/ 2TB SSD) 8
AF-Media : Computing Board Specification Items Description Processor Intel Xeon D-1548 8 Cores/16 Threads 2.0GHz/2.6GHz TDP < 45W Memory RDIMM DDR4-2133, 2Ch 4 Slot up to 128GB No. Item Description No. Item Description Storage OS: msata 64GB SATA3 1 CPU D1540 B NIC Slot OCP Mezz. V2 2 DIMM Slot DDR4 RDIMM C BMC FW BMC Flash 3 JPWR 12V Conn. D vbios FW Video BIOS Flas h 4 SATA SATA3 Conn. E BMC AST2400 5 XDP (CPU) Debug Port F Ethernet Controll er 6 JSPI Front Panel G MGMT RJ45 Intel I210 Network Card Interface OCP Mezzanine V2.0 PCIe 3.0 x8 7 BIOS FW BIOS Flash H Console/USB RJ45, USB2.0 8 msata msata Slot I VGA VGA HDR 9 BAT RTC Battery J FAN FAN HDR A XDP (PCH) Debug Port - Interface for Storage PCIe 3.0 x16 System Management AST2400 9
AF-Media : Storage Module 4x4 SSD Status LED 4x4 SSD Bay SAS & Power Cable Chain OCP Mezz. Network Card Xeon D SoC Raid Controller FAN Global Disk Activity & Status LED Storage Sled Drive Attention LED Cable Chain SAS & Power Cable chain to connect Raid Card and Disk Backplane 10
AF-Media : System Reliability & Availability Node Hot-Plug SSD Hot-Swap Redundant PSU 11
0:00:00 0:00:50 0:01:40 0:02:30 0:03:20 0:04:10 0:05:00 0:05:50 0:06:40 0:07:30 0:08:20 0:09:10 0:10:00 0:10:50 0:11:40 0:12:30 0:13:20 0:14:10 0:15:00 0:15:50 0:16:40 0:17:30 0:18:20 0:19:10 0:20:00 0:20:50 0:21:40 0:22:30 0:23:20 0:24:10 0:25:00 0:25:50 0:26:40 0:27:30 0:28:20 0:29:10 0:30:00 0:30:50 0:31:40 0:32:30 0:33:20 0:34:10 0:35:00 0:35:50 0:36:40 0:37:30 0:38:20 0:39:10 0:40:00 Throhghput(Gbps) AF-Media : Performance Video Streaming Throughput (HLS: Http Live Streaming Protocol) 40.00 35.00 30.00 1Mbps 2Mbps 7Mbps 35Mbps 25.00 20.00 15.00 10.00 5.00 0.00 Time (Sec) 1Mbps (SD) 2Mbps (HD) 7Mbps (Full-HD) 35Mbps (UHD) User Throughput User Throughput User Throughput User Throughput 33.21 36.23 37.37 37.94 26,800 16,000 4,900 925 Gbps Gbps Gbps Gbps 12
AF-Media : Comparison AS-IS AF-Media Form Factor 1U 4U 2U S16 P2 CPU Intel E5 2-Socket Intel Xeon-D Intel Xeon-D Max. Throughput (FTP / RTSP) ~12Gbps / ~10Gbps ~38Gbps / ~25 Gbps ~38Gbps / ~25 Gbps Storage 450GB SAS HDD * 48EA (for 15TB) 1TB SATA SSD * 16ea NVMe PCIe Card 3.2TB * 2ea Power Consumption 880W 150W ( 82%) 130W ( 85%) 13
NV-Array : NVMe SSD Array All-Flash & All-PCIe NVMe SSD Array +4% Baseline Power 0.6 GB/s +2% 1.0 GB/s 2.0 GB/s SATA 6Gb/s PCIe 8Gb/s PCIe 16Gb/s (1 lane) (2 lanes) Two lanes of PCIe 3.0 offers 3.3x the performance of SATA 6Gb/s with only 4% increase in power NVMe SSD Array with PCIe host connection maximizes throughput and latency NVMe SSD moves IO bottleneck from storage to other components in the system The ratio between NVMe SSD and CPU has become very important for efficiency 14
NV-Array : Development Collaboration NV-Array Lightning Form Factor 19 inch 1RU Form Factor 21 inch 2OU Drive Type 2.5 NVMe SSD Drive Type 2.5 & M.2 NVMe SSD Number of SSD 20ea Number of SSD 30ea Raw Capacity 80TB (w/ 4TB NVMe SSD) Re-timer Card PCIe Cable Raw Capacity 120TB (w/ 4TB NVMe SSD) Density SSD Connector 80TB/1RU U.2 (SFF-8639) OpenBMC NVMe SSD... Density SSD Connector 60TB/1OU U.2 & M.2 (SFF-8639) Disk Hot-Plug Support Disk Hot-Plug Support External Cable Mini SAS HD (SFF-8644) External Cable Mini SAS HD (SFF-8644) Number of Hosts Up to 4 Hosts Number of Hosts Up to 4 Hosts Host Connection PCIe 3.0 x8 Host Connection PCIe 3.0 x16, x8, x4 Management OpenBMC Management OpenBMC *This is preliminary specification and subject to change 15
Thank you! 16