Expert Panel: Cloud Storage Initiatives: A Storage Developer Conference Preview August 4, 2015

Transcription

1 Expert Panel: Cloud Storage Initiatives: A Storage Developer Conference Preview August 4, 2015

2 Today s Presenters Alex McDonald, SNIA Cloud Storage Initiative Chair - NetApp Yong Chen Assistant Professor Texas Tech University Mark Carlson Cloud Storage TWG Co-Chair Toshiba David Slik SNIA Cloud Storage TWG Co-Chair NetApp 2

3 SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. 3

4 Storage Developer Conference 2015 Santa Clara, CA September 21-24, 2015 The conference for storage developers since developers, engineers and technical professionals worldwide from the storage community Three focused tracks dedicated to disruptive technologies: Persistent Memory Object Drives SMR Drives Visit 4

5 Mobile and Secure: Cloud Encrypted Objects using CDMI David Slik NetApp, Inc.

6 Data Security in the Cloud Organizations (and individuals) are increasingly concerned about storing unencrypted data in the cloud What happens if the cloud provider is compromised? What happens if the cloud account is compromised? What happens if a system that can access the cloud account is compromised? What happens if the cloud provider goes out of business? All of these scenarios can result in massive data breeches, and are often undetected due to lack of audit 6

7 7

8 What do we need to standardize? How are encrypted objects recognized by clouds? In CDMI, based on a mimetype of application/cms How are requests for plaintext are distinguished from requests for ciphertext? In CDMI, based on HTTP accept header How are requests that require a key made? In CDMI, based on metadata attached to the object How does the cloud determine where to get the key? In CDMI, based on metadata attached to the object 8

9 What do we need to standardize? How does a cloud request a key? New work to standardize a key request message Includes information about the client to allow access control decisions and audit How does the cloud receive a key? New work to standardize a key response message Includes information about how the key can be used, and what is to be passed to the client How is audit requested and managed? 9

10 Why is this important? Allows edge systems to use clouds as untrusted repositories Allows trusted edge clouds to federate with untrusted clouds Allows clients and clouds be able be abstracted from the remote access control provider Allows access control decisions for distributed cloud operations to be locally controlled Allows audit for distributed cloud operations to be locally collected 10

11 Object Drives: A New Architectural Partitioning Mark Carlson Toshiba

12 What are Object Drives? Key/Value semantics (Object store) among others Hosted software in some cases Interface changed from SCSI based to IP based (TCP/IP, HTTP) Channel (FC/SAS/SATA) interconnect moves to Ethernet network 12

13 Object Drive use Interface Key Value example is Kinetic Metadata is not part of the interface Metadata understood by higher levels would be stored as values Not expected to be used directly by end user applications (similar to existing Block I/F in that regard) Higher level example is CDMI Includes rich metadata (both user and system) Expected to be used by end user applications (and is) Object Drives, because of their abstraction allow many more degrees of innovation freedom in the implementations behind this abstraction Hosts needn t manage the mapping of objects to a block storage device An object drive may manage how it places objects on the media without reference to host specified locations 13

14 Types of Object Drives Key Value Protocol (Object Drive) Minimal incremental CPU/Memory requirements Simple mapping to underlying storage In-Storage Compute (Object Drive)* Enough CPU/Memory for Object Node Software to be embedded on the drive General purpose download or factory installed May have additional requirements such as solid state media and more/higher bandwidth networking connections In both cases, the interface abstracts the recording technology *Jim Gray memorial 14

15 Key Value Protocol Object Drives Eliminates existing parts of the usual storage stack Block drivers, logical volume manager, file system And the associated bugs and maintenance costs Existing applications need to be re-written, or adapted Mainly used by green field developed applications Firmware is upgraded as an entire image Hyperscale customers are already doing this Using open source software and creating their own apps Facebook, Google, etc. Key Value organization of data is growing in popularity Examples: Cassandra, NoSQL 15

16 In-Storage Compute Object Drives Same advantages as Key Value protocol plus No need for a separate server to run Object Node service (other services still need a server but scale separately) scaling is smoother only adding drives Additional features of the Object Node software can be deployed independently Fewer hardware types that need to be maintained (for selected use cases) Failure domains are more fine grained, thus overall data availability is enhanced 16

17 Ethernet Connectivity Ethernet speeds standardized at 1Gbps, 10 Gbps Object Drives are high volume, low margin devices 10 Gbps too expensive for the drive form factor for the foreseeable future Desire to have standardized speeds of 2.5 Gbps, 5 Gbps With auto-negotiation among them SFF 8601 effort to specify auto-negotiation using existing silicon implementations (switch firmware upgrade done in a few months) proposed effort to standardize single lane 2.5 and 5 Gbps speeds in silicon (multi-year effort) 17

18 Unistore: A Unified Storage Architecture for Cloud Computing Yong Chen Associate Director, Cloud and Autonomic Computing Site Director, Data-Intensive Scalable Computing Laboratory Assistant Professor, Computer Science Department Texas Tech University (in collaboration with Nimboxx, Inc)

19 Background Enterprise computing apps are highly data intensive Facebook: O(100PB) of storage, with 220M+ photos uploaded/ 25+TB growth per week Google: 1 trillion pages by 2008, 1 billion pages growth per day Info. retrieval, data mining, online business, social network, etc. Scientific computing apps follow the similar trend Scientific computing in Cloud Research as a service Pressure on the storage system capability increases PI Project On-Line Data Off-Line Data Lamb, Don FLASH: Buoyancy-Driven Turbulent Nuclear Burning 75TB 300TB Fischer, Paul Reactor Core Hydrodynamics 2TB 5TB Dean, David Computational Nuclear Structure 4TB 40TB Baker, David Computational Protein Structure 1TB 2TB Worley, Patrick H. Performance Evaluation and Analysis 1TB 1TB Wolverton, Christopher Kinetics and Thermodynamics of Metal and 5TB 100TB Washington, Warren Climate Science 10TB 345TB Tsigelny, Igor Parkinson's Disease 2.5TB 50TB Tang, William Plasma Microturbulence 2TB 10TB Sugar, Robert Lattice QCD 1TB 44TB Source: R. Ross et. al., Argonne National Laboratory 19

20 Traditional Storage Media Over 90% of all data in the world is being stored on magnetic media (hard disk drives, HDDs) IBM invented in 1956 Mechanism remains the same since then Various mechanical moving parts High latency, slow random access performance, unreliable, power hungry Source: online Large capacity, low cost (USD 0.10/GB), impressive sequential access performance 20

21 Emerged Storage Media Non-volatile storage-class memory (SCM) Flash-memory based Solid State Drives (SSDs), PCRAM, NRAM, Use microchips which retain data in nonvolatile memory (array of floating gate transistors isolated by an insulating layer) High throughput, low latency (esp. random accesses), less susceptible to physical shock, power efficient Low capacity, high cost (USD /GB), block erasure, memory wear out (10K-100K P/E cycles) Intel X25-E SSD

22 Storage System Needs and Challenges Conventional HDDs and emerged SCM complement each other well Traditional parallel/distributed file systems designed for HDDs not managing SCM well Straightforward solution not optimal Placing SCM as additional memory hierarchy in the current storage tier Intensive writes go through SCM Inclusive setting limits capacity Do not distinguish criticalness of data 22

23 Our Solution - Unistore: A Unified Storage Architecture A distributed file system store Distinguish workload and access patterns Consider SCM and HDD features Block erasure, internal concurrency, wearleveling High performance, high cost SCM keeps Semantically critical data, e.g. metadata blocks Performance critical data, e.g. frequently accessed High capacity, low cost HDD keeps n Goals p Performance close to SCM devices p Capacity close to the SCM+HDDs p Combines merits and avoids drawbacks Cloud Applications Unistore Low-priority large data sets In-situ processing runtime optimizations 23

24 Unistore: High-level View We are developing a prototype based on Sheepdog, a distributed object storage system for volume and container services Characterization Component Workloads Access patterns Devices Bandwidth Throughput Block erasure Concurrency Wear-leveling Data Placement Component I/O Pattern Random/sequential Read/write Hot/cold Placement Algorithm e.g. modified consistent hashing In-situ Processing Component Active storage Sheepdog architecture Deduplication 24

25 SUORA Algorithm SUORA: Scalable and Uniform storage via Optimally-adaptive and Random number Addressing Divide heterogeneous devices into multi-dimensional space Assigns devices to different segments in each dimension Map data among segments and dimensions Model 25

26 SUORA Algorithm (cont.) SUORA models devices as buckets and segments in numbered lines Different lines (dimensions) represent different factors considered (e.g. throughput, capacity, etc.)! HDD Bucket Node Bucket Capacity Assigned Segments A HDD 1TB HSegment0 (-1, 0) B HDD 0.8TB HSegment3 (-3.8, -3) C HDD 1.5TB HSegment1 (-2, -1) HSegment2 (-2.5, -2) D SSD 0.6TB SSegment0 (0, 1) E SSD 0.3TB SSegment1 (1, 1.5) F SSD 0.8TB SSegment2 (2, 3) SSegment3 (3, 3.3) B (-3.8) (-2.5) HSegment3 SSegment2 HSegment1 HSegment0 SSegment0 SSegment1 SSegment3 SSegment2 C C A D E F F (1.5) (3.3) SSD Bucket 26

27 SUORA Algorithm (cont.) Initial comparison with CRUSH, consistent hashing, ASURA different algorithms In terms of computation time (of data distribution), memory footprint, uniform distribution, adaptation to device additions and removal The initial comparison has shown the promise of SUORA algorithm in Unistore architecture More details will be in the SNIA SDC presentation 27

28 Summary Unistore is an ongoing effort of attempting to build a unified storage architecture for Cloud storage system With the co-existence and efficient integration of heterogeneous HDDs and SCM devices An initial prototype being developed based on Sheepdog A SUORA algorithm is being developed to manage heterogeneous devices in a multi-dimensional space Thank You! Please visit:

29 Using CDMI to Manage Swift, S3, and Ceph Object Repositories David Slik NetApp, Inc.

30 Converged Data Management Objects, Files, LUNs, NoSQL databases, and many other types of persistent data entities can all be managed through the Cloud Data Management Interface (CDMI) This presentation focuses on how CDMI provides a common management interface that sits beside the predominant data access interfaces, such as NFS, CIFS, iscsi, S3, Swift, etc. This is especially valuable for converged storage infrastructure that allows data objects to move between platforms and interfaces 30

31 CDMI Namespaces CDMI supports arbitrary tree-based hierarchies CDMI references allow symlink like cross-tree links Superset of Swift, S3, Azure and other cloud namespaces Superset of standard filesystem namespaces Superset of LUN/Zone hierarchies This allows a single CDMI namespace to encapsulate all of these different data types into a single namespace for management purposes: 31

32 Varied Namespace Restrictions CDMI Object Model Overlay for Swift Object Model Root Container System Capabilities CDMI Root Root Container Container Container Capabilities Account CDMI Container (Representing Account) Data Object (or Queue) Data Object Capabilities / Requirements Container CDMI Container Object CDMI Data Object Overlay for File System Model Overlay for S3 Object Model Root CDMI Directory Root Container CDMI Root Root Container Directory CDMI Container Bucket CDMI Container Account File CDMI Data Object Object CDMI Data Object 32

33 After This Webcast This webcast and a copy of the slides will be posted to the SNIA-CSI website and available on-demand A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA Cloud blog Follow us on Google Groups: 33

34 Questions? Thank You Please rate this talk! 34