Cloud Open Source Innovation on Software Defined Storage

NorthEast ASIA OSS Promotion Forum Cloud Open Source Innovation on Software Defined Storage Hiroshi Miura Director of Japan OSS Promotion Forum OSS Cloud Evangelist, NTT DATA Corporation. Copyright 2014 NTT DATA Corporation

2 Agenda 1.Cloud and Open Source Software 2.Swift 3.Sheepdog 4.Future innovations

3 Cloud and Open Source Software There are many OSS for Cloud in wild

4 Trend on Cloud and OSS Operating Environment OpenStack: Linux for Cloud Docker: PaaS enabler Devops Puppet/Chef/Ansible Automation Kubernertus Software Defined Infrastructure

Software Defined Infrastructure Software Defined Network Open Flow OpenDelight Software Defined Storage OpenStack Swift Ceph/GlusterFS/Sheepdog Software Defined Datacenter Apache mesos Copyright 2014 NTT DATA Corporation 5

6 Swift

Growth of Data Data Explosion on Enterprise Amount of Unstructured Data has been growing. We need storage with Scalability, Durability, Availability. Amount of Unstructured Data EB or PB scale Growing exponentially Examples of Unstructured Data Media (Images, Videos, Audios) Web Contents Documents Backups/Archives 2004 2007 2010 2013 2016 Where should we store these data? One of the Solutions is Swift. Copyright 2014 NTT DATA Corporation 7

Cloud solution for Storage: Swift Swift is... A storage system with Scalability, Durability, Availability. The REST-ful Distributed Object Storage likely Amazon S3. One of OpenStack Core Components. Implemented by Python. A Open Source Software. 1 Block Storage (Cinder) 2 Object Storage (Swift) Copyright 2014 NTT DATA Corporation 8

9 Swift as a storage for a variety of applications System Backup CMS FTP-like use Digital Distribution Web Apps Cyber Duck REST API Swift

10 OpenStack Swift deployments and use cases Name of enterprise Product/ service Description Rackspace(USA) Korean Telecom (Sourth Korea) Cloud Files ucloud storage service Cloud file share service by Rackspace itself. They use same code as OSS except for features such as authentication, Accounting and CDN (<500PB) Object storage service using OpenStack/Swift (16PB+ size) Sina (Republic of China) Sina App Engine(SAE) Public storage service. They moved to OpenStack from another technology MongoDB in 2012. San Diego Supercomputer Center (USA) SDSC Cloud Storage Services Cloud storage service on SDSC. Users can select Amazon/S3 or Rackspace Swift. SME Storage (USA) SMEStorage Open Cloud Platform Cloud storage service based on Rackspace Cloud File IBM/SoftLayer (USA) SoftLayer Object Storage Public object storage service. Acquisition by IBM NTT DATA (Japan) Private Cloud Storage Provide professional service and Operation and management product HP(USA) HP Cloud Private cloud storage service uses OpenStack. Wikimedia(USA) Wikimedia storage Media files store for Wikipedia. NII(JAPAN) Academic Cloud service Academic cloud service by National Institute of Informatics in Japan (NII)(Integrated and supported by NTT Data)

11 Architecture: Nodes Swift consist of 2-type Nodes: Proxy Node and Storage Node. Application Forward Data to node HTTP Load balancer Proxy Node Proxy Node Proxy Node Store data Storage Node Storage Node Storage Node Storage Node

12 Architecture: The Ring The Ring (static table for data allocation on storage node) decide the optimal Storage Node by Name. Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Ring Ring Ring Ring

13 Architecture: The role of Ring If you requested to Store the data A, 3 Replica nodes store the data A. data Application HTTP Load balancer Proxy Node Proxy Node Proxy Node A must be located at 1, 2, 4 Ring Ring Ring Storage Node 1 Storage Node 2 Storage Node 3 Storage Node 4 Data A Data A Data B Data A Data B Data B Ring Ring Ring Ring

14 Architecture: The role of Ring If you requested to Get the data A, One of Nodes reply the data A. data Application data HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring A must be located at 1, 2, 4 Storage Node 1 Storage Node 2 Storage Node 3 Storage Node 4 Data A Data A Data B Data A Data B Data B Ring Ring Ring Ring

15 Scalability (1) Expand proxy server Throughput (2)Expand Storage servers or disks volume More Throughput Proxy Proxy (expand) Proxy Proxy Storage Storage Storage Storage Storage Storage Storage (Expand) More Volume

16 Our deployment

17 Our Project Migration from existing high-end storage to distributed storage Moves rarely accessed data from expensive high speed storage to Swift storage Backend Storage for career application which stores more than 1,000,000 user s data End Users Application server High-end storage Data migration Swift Distributed Storage Expansion depends on growth Self healing Disaster recovery

18 Automated Building and testing Automated Building Automated Testing puppet Kickstart install & configure Swift Swift OS OS swift proxy servers Test all APIs Test all storage nodes via proxy Tempest dispersionpopulate ex. swift proxy servers configuration server Swift OS Swift OS swift storage servers Automated building tools for swift Install OS and rpms with Kickstart Configure all software with puppet swift storage servers Automated testing tools of swift cluster Test all APIs with tempest Test all storage nodes with original tool based on swift-dispersion-populate Speedup of swift cluster building We built more than 100 nodes in 1 day! Quality assurance by eliminating human error All configurations are set automatically! Speedup of swift cluster testing Realize quality assurance of all nodes All swift nodes are tested!

NTT DATA Open Solutions - Block Storage NTT Data Open Solutions Block Storage bundle commodity Intel architecture servers in single cluster build an unified distributed network storage. Provide iscsi interface for Windows and Linux Servers High scalability than traditional storage products AP Management mount Storage Pool NTT DATA Open Solutions Block Storage Copyright 2014 NTT DATA Corporation 20

21 Features 1. Flexible and effective scale in sizes capacity 2. Self healing made reduce operation cost NTT DATA Block Storage Applic aiton iscsi Sheepdog clusters Visualize tool Management tool 3. Easy to use 4. Integrated management

22 Future Innovations

SDS brings the day all data digitalized These Software Defined Storage and Cloud Technology leads All data can be stored in the storage All type of data can be stored All Data and All type of data Machine generated data Privacy reduced personal data Office data Accelerate Big Data analysis Integrated into Software Defined Datacenter With Software defined network Next generation of XaaS(Self-service) Full Automation / Self-healing / Scaling Copyright 2014 NTT DATA Corporation 23

24 Our business

25 Our team and products Hinemos Integrated operation and management software GresCube Appliance for PostgreSQL Talents OSS Support service OSS professional team help you PRORIZE High quality infrastructure builder Teams Services Hadoop Professional service Japanese No1. team build Hadoop clusters Virtual Network Controller Freedom of Network With OpenFlow Full-Open IaaS Build OSS with OSS Private Cloud Storage Peta-byte class private storage