Design and Performance Evaluation of Networked Storage Architectures

Transcription

1 Design and Performance Evaluation of Networked Storage Architectures Xubin He July 25,2002 Dept. of Electrical and Computer Engineering University of Rhode Island

2 Outline Introduction STICS: SCSI-To-IP Cache for Storage Area Networks DRALIC: Distributed RAID & Location Independence Cache vcraid: Large Virtual NVRAM Cache for Software RAID Performance Eval. on Distributed Web Server Architectures Conclusions

3 Background Data storage plays an essential role in today s fast-growing data-intensive network services. Online data storage doubles every 9 months Storage is approaching more than 50% of IT spending.the storage cost will be up to 75% of the total IT cost in year 2003.

4 A Server-to-Storage Bottleneck Source: Brocade

5 Motivations How to deploy data over the network efficiently and reliably? Disparities between SCSI & IP STICS SCSI remote handshaking over IP Processor-disk gap growing High speed network DRALIC Large client memories Cheap Disk & RAM, expensive NVRAM RAID5 is reliable, but low performance E-commerce over the Internet, distributed web servers vcraid

6 Introduction STICS: SCSI-To-IP Cache for Storage Area Networks DRALIC: Distributed RAID & Location Independence Cache vcraid: Large Virtual NVRAM Cache for Software RAID Performance Eval. on Distributed Web Server Architectures Conclusions

7 Introducing a New Device:STICS Whenever there is a disparity, cache helps Features of STICS: Smooth out disparities between SCSI and IP Localize SCSI protocol and filter out unnecessary traffic reducing bandwidth requirement Nonvolatile data caching Improve performance, reliability, manageability and scalability over current iscsi systems.

8 System Overview Host 1 SCSI NAS Disks or SAN STICS 1 TCP/IP TCP/IP Internet TCP/IP SCSI STICS 2 Host 2 or Storage SCSI STICS 3 STICS N SCSI Host M or Storage System overview. A STICS connects to the host via SCSI interface and connects to other STICS or NAS via Internet.

9 STICS Architecture SCSI Interface Log Disk RAM Processor Storage device Network Interface

10 Internal Cache Structure Memory Meta Data Cache Data Cache log Disk

11 Basic Operations Write Write requests from the host via SCSI Write requests from another STICS via NIC Read Read requests from the host via SCSI Read requests from another STICS via NIC Destage RAM > log disk Log disk > storage device Prefetch Storage device > RAM

12 Web-based Network Management Web browser-based Manager HTTP HTTP Servlet Management App. TCP/IP TCP/IP Local Manage App.

13 Implementation Platform A STICS block is a PC running Linux OS: Linux with kernel Compiler: Interfaces: gcc SCSI STICS IP

14 Performance Evaluations Methodology iscsi implementation on Linux by Intel (iscsi) Initial STICS Implementation on Linux Two modes: Immediate report (STICS-Imm) Report after complete (STICS) Workloads Postmark of Network Appliances: throughput Two configurations Small: 1000/50k/436MB Large: 20k/100k/740MB EMC Trace :response time More than 230,000 I/O requests Data set size: >900MB

15 Experimental Settings Host (Trout) NIC iscsi commands and data Switch NIC Target (Squid) SCSI Disks Cod iscsi configuration. The host Trout establishes connection to target, and the target Squid responds and connects. Then the Squid exports hard drive and Trout sees the disks as local. Host (Trout) Block Data STICS 1 Switch STICS 2 Target (Squid) SCSI Disks Cod STICS configuration. The STICS cache data from both SCSI and network.

16 PostMark Results: Throughput Throughput (1000 initial files and 50k transactions) STICS-Imm STICS iscsi Throughput (20k initial files and 100k transactions) STICS-Imm STICS iscsi Transactions/sec Transactions/sec Block size (bytes) Block size (bytes) Ave. Improvement Small set Large set STICS-imm 226% 318% STICS 64% 97%

17 Where does the benefit come from? < >1024 iscsi 7 1,937, ,415,912 STICS 4 431, ,827 # Of packets with different sizes (bytes) Total Packets Small Packets (%) Bytes Transferred Bytes per packet iscsi 3,353, % 1,914,566, STICS 1039, % 980,963, Network traffic analysis

18 EMC Trace Results: Response Time a) STICS with immediate report(2.7 ms) b) STICS with report after complete (5.71 ms). c) iscsi (16.73 ms). Histograms of I/O response times for trace EMC-tel.

19 Summary A novel cache storage device that adds a new dimension to networked storages Significantly improving performance of iscsi A cost-effective solution for building efficient SAN over IP Allow easy manageability, maintainability, and scalability

20 Introduction STICS: SCSI-To-IP Cache for Storage Area Networks DRALIC: Distributed RAID and Location Independence Cache vcraid: Large Virtual NVRAM Cache for Software RAID Performance Eval. on Distributed Web Server Architectures Conclusions

21 Web Servers Overhead caused by FS is high Enterprise web server is expensive A Fujitsu Server: More than $5 million PCs are cheap: $1000 Disks: $160/120GB (IBM Deskstar@CompUSA) DRAM:$100/256MB(@Crucial.com)

22 My Solution Combine or bridge the disk controller and network controller of existing PCs interconnected by a high-speed switch. Share memory and storage among peers

23 D B M S D B M S F i le S y s t e m F i le S y s t e m D is k D r iv e r B r id g e N e tw o r k D r iv e r S C S I N IC N e tw o r k D r iv e r N IC B r id g e D is k D r iv e r S C S I D S M R A M R A M R A P I D F a s t L A N (S w itc h ) R A M R A M S C S I D is k D r iv e r B r id g e N IC N e tw o r k D r iv e r N IC S C S I N e tw o r k D r iv e r B r id g e D is k D r iv e r F i le S y s t e m F i le S y s t e m D B M S D B M S

24 Performance analysis T T T T DRALIC lm rm RAID = H B = BW B = BW mem net lm T ( N 1) lm + OH B = N BW net + net + ( 1 H ) H T + ( 1 H )( 1 H ) + lm B BW mem rm rm B N BW ( N 1) OHnet + + OHdsk dsk lm rm T RAID B: data block size (8KB) N: number of nodes H lm : Local memory hit ratio H rm : Remote memory hit ratio T lm : Local memory access time T rm : Remote memory access time T raid : access time from the distributed RAID T dralic : Average response time of DRALIC system

25 Preliminary Performance Analysis 7 DRALIC: Nodes infl uence Hlm=0.5 Hlm=0.8 Access time(ms) Hlm= Hlm= Nodes Average I/O response time vs. number of nodes

26 Simulation Results DRALICSim: a simulator based on socket communication. Benchmark: PostMark: measures performance in terms of transaction rates provided by Network Appliance Inc. Configurations: 1000 initial files and transactions (small), 20000/50000(medium) and 20000/100000(large) 4 Nodes running Windows NT

27 Simulation Results Throughput Base 2Nodes 3Nodes 4Nodes Transactions/Sec Small Medium Large Test Suite

28 Summary Combination of HBAs and NICs will reduce the overhead. Share memory and storage among peers Make use of existing resources Our simulator has the performance gain up to 4.2 with 4 nodes

29 Introduction STICS: SCSI-To-IP Cache for Storage Area Networks DRALIC: Distributed RAID & Location Independence Cache vcraid: Large Virtual NVRAM Cache for Software RAID Performance Eval. on Distributed Web Server Architectures Conclusions

30 VC-RAID Hiding the small write penalty of RAID5 by buffering small writes and destaging data back to RAID with parity computation when disk activity is low. A combination of a small portion of the system RAM and a log disk to form a hierarchical cache. This hierarchical cache appearing to the host as a large nonvolatile RAM.

31 Architecture Main Memory Buffer Cache OS kernel Cache Disk RAID5

32 Approaches RAM buff er RAM buff er Controller Cache Disk RAID5 Controller Cache Disks RAID5 (a) (b) RAM buff er Controller Cache Disks Controller RAM buffers Cache Disks (c) RAID5 (d) RAID5

33 Performance Results Test environment: Gateway G6-400, 64MB RAM, 4M RAM buffer, 200 MB Cache disk, 4 SCSI disks form a disk array. Benchmarks Postmark by Network Appliance Untar/copy/remove Compared to built-in RAID0 and RAID5

34 Throughput Series RAID 0 VC-RAID RAID 5 Small (1k+50k) Medium (20k+50k) Large (20k+100k)

35 untar S yn chr on ous Asyn chr on ous Remove Synchr onous Asynchr onous vc-raid raid5 raid Copy Synchr onous Asynchr onous 0 vc-raid raid5 raid Response time (second) vc-raid raid5 raid0

36 Summary Reliable: based on RAID5 Hard drive is more reliable than RAM Cost effective: hard drives are much cheaper than RAM Software, don t need extra hardware Fast: increasing the cache size

37 Introduction STICS: SCSI-To-IP Cache for Storage Area Networks DRALIC: Distributed RAID & Location Independence Cache vcraid: Large Virtual NVRAM Cache for Software RAID Performance Eval. on Distributed Web Server Architectures Conclusions

38 Observations E-Commerce has grown explosively Static web pages that are stored as files are no longer the dominant web accesses. about 70% of them start CGI, ASP, or Servlet calls to generate dynamic pages. Web server behaviors and the interaction between web server and database servers

39 W S (a ) DBS W S DBS1 (b ) Pro xy (W S Selector) (c ) W S 1 W S 2 D B S 1 Pro xy (W S Selector) W S 1 DBS1 W S 2 (d ) DBS2 W S 1 Pro xy (W S Selector) W S 2 D B S 2 (e ) W S 3 D B S 1

40 Benchmark and workloads Workloads Static pages Light CGI: 20% / 80%. Heavy CGI: 90% / 10%. Heavy servlet: 90% / 10%. Heavy database access: 90% /10%. Mixed workload: 7% / 8% / 30% /55% WebBench 3.5 (6010 static pages, 300 cgi, 300 simple servlets, 400 DB servlets using JDBC, 2 databases with 15 and 18 tables)

41 Throughput(Static Pages) Throughput (Light CGI) 1ws 2ws 3ws 1ws 2ws 3ws Req/Sec Clients Req/Sec Clients Throughput(Heavy CGI) 1ws 2ws 3ws 800 Throughput(Heavy Servlet) 1ws 2ws 3ws Req/Sec Req/Sec Clients Figure 5(a) Clients

42 CPU time distribution(pc Server3) cpu time distribution (PC Server3) idle user kernel ilde user kernel cpu time% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% cgi runs on apache clients cpu time% 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% servlet runs on java web server clients Throughput (Heavy Database access) Throughput (Mixed workload) 1wsdbs 1ws1dbs 2ws1dbs 2ws2dbs 3s2dbs 1wsdbs 1ws1dbs 2ws1dbs 2ws2dbs 3ws2dbs Req/Sec Figure 7(a) Clients Req/Sec Clients

43 Introduction STICS: SCSI-To-IP Cache for Storage Area Networks DRALIC: Distributed RAID & Location Independence Cache vcraid: Large Virtual NVRAM Cache for Software RAID Performance Eval. on Distributed Web Server Architectures Conclusions

44 Summary STICS couples reliable and high speed data caching with low overhead conversion between SCSI and IP. DRALIC boosts the web server performance by combining disk controller and NIC to reduce FS overhead. vcraid presents a reliable and inexpensive solution for data storage. We carried out an extensive performance study on distributed web server architectures under realistic workloads.

45 Patents (with Dr. Yang) STICS: SCSI-To-IP Cache Storage, File pending, Serial Number 60/312,471, August 2001 DRALIC: Distributed RAid and Location Independence Cache, Filed pending, May 2001

46 Publications (Journal) 1. Xubin He, Qing Yang, and Ming Zhang, STICS: SCSI-To-IP Cache for Storage Area Networks, Submitted to IEEE Transactions on Parallel and Distributed Systems. 2. Xubin He, Qing Yang, Performance Evaluation of Distributed Web Server Architectures under E- Commerce Workloads, Submitted to Journal of Parallel and Distributed Computing. 3. Xubin He, Qing Yang, On Design and Implementation of a Large Virtual NVRAM Cache for Software RAID, Special Issue of Journal on Parallel I/O for Cluster Computing, 2002.

47 Publications (Conference) 1. Xubin He, Qing Yang, and Ming Zhang, A Caching Strategy to Improve iscsi Performance, To appear in IEEE Annual Conference on Local Computer Networks, Nov. 6-8, Xubin He, Qing Yang, and Ming Zhang, Introducing SCSI-To-IP Cache for Storage Area Networks, ICPP 2002, Vancouver, Canada, August Xubin He, Ming Zhang, Qing Yang, DRALIC: A Peer-to-Peer Storage Architecture, Proc. of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'2001), Xubin He, Qing Yang, Characterizing the Home Pages, Proc. of the 2nd International Conference on Internet Computing (IC 2001), Xubin He, Qing Yang, VC-RAID: A Large Virtual NVRAM Cache for Software Do-it-yourself RAID, Proc. of the International Symposium on Information Systems and Engineering (ISE'2001), Xubin He, Qing Yang, Performance Evaluation of Distributed Web Server Architectures under E-Commerce Workloads, Proc. of the 1 st International Conference on Internet Computing (IC 2000), 2000.

48 Thank You! Dr. Qing Dr. Jien-Chung Dr. Joan Dr. Peter Dr. Lisa And more

49 Special thanks to my daughter, Rachel!

50

51

52