HADP Talk BlueDBM: An appliance for Big Data Analytics

Transcription

1 HADP Talk BlueDBM: An appliance for Big Data Analytics Sang-Woo Jun* Ming Liu* Sungjin Lee* Jamey Hicks+ John Ankcorn+ Myron King+ Shuotao Xu* Arvind* *MIT Computer Science and Artificial Intelligence Laboratory +Quanta Research Cambridge Presented by: Geethan Karunaratne Supervision: Gustavo Alonso 1

2 Big Data Success Stories Google predicts Flu outbreaks image: 2

3 Big Data Success Stories Amazon anticipatory shipment of products image: 3

4 Big Data Success Stories Genome analysis and personalized diagnostics image: 4

5 Big Data : Future Likely to be the biggest economic driver for the IT industry for the next decade. Predictive Analysis, Identify issues and or make recommendations Banking Business Finance Communication Health Data-as-a-Service Algorithm markets (Algorithmia, Data Xu, and Kaggle) Data in the range 5TB to 20 TB range: continue to grow Right appliance for big data processing? 5

6 Memory/Storage Technologies Power HDD Latency 5 us SSD Latency us Latency Cost per bit 6

7 A currently popular solution: RAM Cloud[1] Cluster of machines with large capacity and fast interconnect + Fastest as long as data fits in - Power hungry and expensive - Performance drops when data doesn t fit in What if enough isn t affordable? -based solutions a more appealing alternative + Faster than Disk, cheaper than + Lower power consumption than both - Legacy storage access interface is burdening - Slower than 7

8 Latency profile of distributed flash-based analytics Distributed processing involves many system components device access Storage software (OS, FTL, ) Network interface (10gE, Infiniband, ) Actual processing Access 75 μs Storage 100 μs Network 20 μs Processing 50~500 μs 100~1000 μs 20~1000 μs Latency is additive 8

9 Latency profile of distributed flash-based analytics Architectural modifications can remove unnecessary overhead Cross-layer optimization of flash management software Dedicated storage area network Near-storage processing accelerator Access 75 μs Storage Network Network Processing Processing 20 μs 20 μs 100 μs 50~100 μs <20μs 20~ ~1000 μs μs 20~1000 μs Processing 9

10 Latency profile of distributed flash-based analytics Architectural modifications can remove unnecessary overhead Access 75 μs Storage 100 μs 50~100 μs 100~1000 μs 1 Network 20 μs Processing 20~1000 μs 2 3 Access 75 μs 50~100 μs <20μs 10

11 Related work 1. Use of flash FusionIO, Purestorage, Moneta, Willow SSD for database buffer pool and metadata Zetascale Biscuit (ISCA 2016) Netezza [SIGMOD 2008], [IJCA 2013] 2. Networks QuickSAN [ISCA 2013] Hadoop/Spark on Infiniband RDMA [SC 2012] 3. Accelerators SmartSSD[SIGMOD 2013], Ibex[VLDB 2014] Catapult[ISCA 2014] Centaur [FCCM 2017] Biscuit (ISCA 2016) Netezza (IBM) GPUs 11

12 BlueDBM: Overall Architecture 1 of 2 Racks (10 Nodes) 12

13 Overall Architecture core Xeon Servers 20 1TB flash storage 20 Custom flash boards 20 Xilinx VC707 PCIe 1 Gen 10 Gbps controller network 13

14 Node Architecture 14

15 Node Architecture 15

16 Use of 16

17 ISP/Host HPC FMC PORT Custom flash card 512GB Cont. Bus 0 Bus 1 Bus 2 Bus 3 Network Ports 17

18 To VC707 HPC FMC PORT Custom flash card Limited program cycles in flash CPU for Artix 7 FPGA Bus 0 Bus 1 Bus 2 Bus 3 Network Ports Wear leveling Garbage collection Bit error correction Bad lock management In BlueDBM, these are offloaded to filesystem and block devices drivers 18

19 Custom flash card High Performance Access Handle out of order and Interleaved ( Server) Need for reorder buffer in host side Multi Agent access ( Interface Splitter) Local host Local in-store processor Remote in-store processor 19

20 Near Storage Processing 20

21 In-store Processing Engine User defined in-store processing engines Access to Interface Host Interface Network Interface Key in sending /receiving data to/from remote storage Implemented on Virtex 7 FPGA 21

22 Userspace Interface Hardware-assisted Applications Connectal Proxy Kernelspace File System Generated by Connectal* Block Device Driver FPGA Connectal (By Quanta) Ctrl HW Accelerator Connectal Wrapper Accelerator Manager Network Interface NAND BlueDBM provides a generic file system interface as well as an accelerator-specific interface (Aided by Connectal) 22

23 Networks 23

24 Integrated Storage Network Packet-switched mesh network connected by high performance serial links (SATA) 24

25 Integrated Storage Network Flow control and virtual channel Deterministic routing Network configuration file to populate the routing tables 25

26 Integrated Storage Network Virtex Artix Host Interface ISP Network Interface Aurora 4-Lanes 26Gbps 0.5us 4x GTX Transceivers 10Gbps, 0.5 us SATA Fanout 8 4x GTP Transceivers 6.6Gbps, 0.5 us Questions device selection Fanout 8 means upto 64 nodes reached within 2 hops 26

27 Evaluation 27

28 Evaluation FPGA resource utilization Module Name LUT Registers BRAM Module Name LUT Registers BRAM Bus s Interface SerDes Network Interface Total 56% 23% 50% Interface Host Interface Total 45% 22% 22% Artix 7 Virtex 7 28

29 Power Consumption Component Power (Watts) VC Board (x2) 10 Storage Device Total 40 Power Consumption: TeraData Aster 50x RamCloud 4x Component Power (Watts) Storage Device 40 Xeon Server 200+ Node Total

30 Access Bandwidth 3.3 GB/s 2.4 GB/s Int. Switch NIC In-store Proc PCI 1.6 GB/s 1 GB/s or 0.66 GB/s 30

31 Access Bandwidth 3.3 GB/s 2.4 GB/s Int. Switch In-store Proc NIC 1 GB/s or 0.66 GB/s PCI 1.6 GB/s 31

32 Access Bandwidth 3.3 GB/s 2.4 GB/s Int. Switch In-store Proc NIC 1 GB/s or 0.66 GB/s PCI 1.6 GB/s 32

33 Access Bandwidth 3.3 GB/s 2.4 GB/s Int. Switch In-store Proc NIC 1 GB/s or 0.66 GB/s PCI 1.6 GB/s 33

34 Latency Profile of a Node 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us 34

35 Remote Storage Access Latency From in-store processor to remote flash storage us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 35

36 Remote Storage Access Latency From host server to remote flash storage us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 36

37 Remote Storage Access Latency From host server to remote flash storage via its host server us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 37

38 Remote Storage Access Latency From host server to remote us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 38

39 Latency to different memories Near-uniform latency access? us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us (1%) us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us (1%) us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 39

40 Latency to different memories Near-uniform latency access? us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us (1%) us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 0.5 us (1%) us 0 us Int. Switch NIC In-store Proc 5 us PCI 50 us 40

41 Application Acceleration Nearest Neighbour Search 41

42 Application Acceleration Nearest Neighbour Search 42

43 Application Acceleration Nearest Neighbour Search 43

44 Application Acceleration Nearest Neighbour Search 44

45 Conclusion and Remarks Presents a complete end to end system More affordable and less-power consuming Demonstrated on hardware Several algorithms tested. Plans to test more. Broad range of contributions for a single paper 45

46 Conclusion and Remarks Near-uniform latency claim questionable Hardware selection could be improved (going for smaller single FPGA): To make small form factor Improve worst case network performance Reduce cost ASIC? 46

47 Thank you 47

48 References [1] [2] sh+memory+and+other+technologies 48

49 Big Data Success Stories Twitter (Social Media) reveal social upheavals image: