Enabling Data Control in a Multi-Cloud World

Transcription

1 Zenko Webinar: Giorgio Regni Scality Co-founder & CTO Laure Vergeron Software Engineer Enabling Data Control in a Multi-Cloud World Thursday, August 3, 2017

2 Zenko Webinar: Multi-cloud, hybrid stores, open-source We ll shed some light on a few questions: What does multicloud mean? How to leverage the efficiency of both public and private clouds? How can this multi-cloud data controller do search and discovery across clouds? How can you get involved with Zenko? 1

3 What does multicloud mean? Freedom to leverage multiple, different cloud infrastructures, private or public Acknowledge that each application has its own infrastructure requirements that will evolve over time Acknowledge that each cloud service has their own domain of expertise and leverage the native services they offer 2

4 Examples of Use-Cases For Multi-Cloud? To you, what does multi-cloud enable? (here is one of our answers) 3

5 Examples of Use-Cases for Multi-Cloud Content Distribution Media companies have tens of thousands of movies, which they store on Private Cloud for control. When it is time to publish a movie, it makes sense to copy it to a public cloud to use its transcoding and CDN services. Compute Bursting Banks have to do risk analysis leveraging thousands of CPU every night. These intense computation only run for a few hours. Rather than having idle servers for the rest of the day, it makes sense to use Public Cloud services for the computation Analytics E-commerce company do more and more machine learning on their very large data lake. Rather than setting up Hadoop infrastructures in-house, the company can copy just a data set to an Hadoop cloud, compute the appropriate algorithm, and get back the result and destroy the cloud copy of the data to save on storage cost. Long-term Archival / cold storage While storing data which is regularly accessed is cheaper in a private cloud, long term archive of never accessed data is cheaper in long term archive cloud offering. Automatic archival of never accessed data would save a lot of money.

6 The Zenko Vision Control and Freedom for data in a Multi-Cloud world. 5

7 The Zenko Multi-Cloud Data Controller Single Interface to any Cloud S3 API as a single API set to any cloud Allow reuse in the Cloud One S3 API for any cloud Maintain the native cloud format Always know your data and where it is Native format Data Insight Metadata search Trigger actions based on data Data Management Data Workflow to manage replication, location 6

8 The Zenko Multi-Cloud Data Controller Allow reuse in the Cloud Maintain the native cloud format Full compatibility with S3 IAM, policies, request syntax... 7

9 Zenko is not Scality s first open-source project... What was Zenko s CloudServer previous name? (check your answer on DockerHub) 8

10 Open Source Scality CloudServer Adoption Now Over 700,000 Launched June 2016 Open-source implementation of AWS S3 API Code available on Github under Apache 2.0 license Packaged in Docker container for easy deployment Seamless upgrade to S3 Connector for the RING

11 Developers Are Seeing the Benefits "Scality provides our backend storage and gives us a single interface for developers to code within any cloud on a common API set. With Scality, we can write an application once and deploy anywhere on any cloud. Mathias Herberts, co-founder and CTO at Cityzen Data We are big users of Docker in our production environment and implemented the Docker version of Scality S3 Server. It is efficient, secure with encryption and S3 authentication, and very easy to maintain. Christian Patry, System Engineer, BlueSolutions by Polyconseil 10

12 CloudServer Amazon S3 & Native Format Amazon S3 API is the defacto standard Extended to support multiple cloud backends Provides a full featured S3 interface independent of the backend cloud stores ability Native Cloud Data Format Data stored in cloud must be accessible in standard format Access via standard keys and methods - enables use by cloud services without change Highly Available Cloud Service Integrated in HA services through Docker If higher level of availability required then extensible to RING S3 API CLOUD GATEWAY OPAQUE DATA?? Many gateway products store data in a closed "black box" format that is unreadable by native cloud services and apps in the cloud. Gateways store "opaque data" in the cloud as a black box. Cloud apps try to access it, but cannot due to the proprietary format. 11

13 Do you know your AWS basics? What is the default location constraint for AWS S3? (AWS S3 documentation answers...) 12

14 Native Cloud Data & Extended Location Control Zenko Manages Bucket Metadata Namespace Decoupled from the underlying data location Native APIs used to store data METADATA: Zenko Namespace PUT Bucket1 LocationConstraint: RING-West PUT Bucket2 LocationConstraint: S3-US-East-1 PUT Bucket3 LocationConstraint: Azure-US Control of Buckets location Provides the default storage location for objects stored (PUT) into that Bucket Buckets can be managed across multiple RINGs & Public Cloud regions Location Mapping is managed through a configuration file Innovation: Extended Location Control across clouds DATA: via native drivers/apis REST/Sproxyd Location: RING-West S3 API Location: us-east-1 Blob Storage API Location: Windows.AzureStorage.US 13

15 Backbeat - Policy-Based Replication Across Clouds Replication supports copy of objects across RINGs Externalized via the AWS S3 API for Cross-Region Replication (CRR) Configure replication on a source Bucket and assign the Target Bucket in any cloud Objects are asynchronously replicated to target with native key/data format Zenko Namespace AWS S3 Namespace Amazon CloudFront Backbeat (Async Replication) S3 BUCKET CRR API PUT PUT Amazon EC2 OBJ1 RING1 OBJ1 S3-us-east-1 Amazon EMR 14

16 Backbeat Architecture (Details) 15

17 Clueso - Metadata Search Federated Search on Metadata Search across cloud namespaces independent of location Applications can attach extended metadata as key/value pairs through optional S3 x-amz-meta- headers Search on one or multiple attributes, fuzzy searches Examples: SELECT key where ContentType = PDF SELECT key where Title= Mathematics% Clueso Search Engine Programmatic Access to Search Accessed through RESTful API with attribute parameters Natural fit with S3 semantics, e.g GET /bucketname?search&attributekey= attributevalue 16

18 Clueso - Overview and Goals Goals are to allow user to retrieve keys: By user metadata (x-amz-meta headers and tags) which are not predefined By object owner Created before or after some date, or range of dates Using multiple attributes using AND and OR conditions Fuzzy search on attribute values, such as partial strings and wildcards such as with SQL LIKE statements Programmatic access to search functionality via an API

19 Why Spark? Distributed Search by design SQL Semantics Not a separate database No need to pre-index (Juliet can create new attributes on the fly) No need to store separate indexes Fast Processing - 100x faster than Hadoop Flexible Engine Can use same Spark cluster to do Athena-like search on the actual data Largest Open Source community in big data - over 1000 contributors Ecosystem of contributors and companies 18

20 Open Source Code on Github under Apache 2.0 License 19

21 Zenko Open Source: Features & Capabilities Zenko Open Source APP APP S3 CALLS S3 CALLS S3 API Single API set and 360 access to any cloud DATA S3 API S3 API Bucket LOCATION Bucket LOCATION METADATA CRR/DATA METADATA DATA CLUESO Metadata Search BACKBEAT Data Policy Engine Native format Data written through Zenko is stored in the native format of the target cloud storage and can be read directly, without going through Zenko. Project Backbeat for data workflow Policy-based data management engine METADATA DATA STORAGE DMD Shared Local Storage REST/Sproxyd AWS S3 API AZURE BLOB API Project Clueso for metadata search Apache Spark-based metadata search tool for optimal data insight HA/Failover Deployed as dual-containers managed by Docker Swarm for HA, but not full scale-out Simple Security single-tenant credentials managed locally

22 Ecosystem Extensions: Community & Partner Driven Data Storage Back-ends - Existing interest in integration of NAS filers - Other public clouds: Oracle, Backblaze and OpenStack based Clueso Search Plugins - GDPR discovery (find data that is not compliant) - Data analytics - ediscovery for legal documents Backbeat plugins for Data Management & Mobility - Migration - Compliance 21

23 Building a Developer community Community Meetups Initiated prior to our S3 Server launch Participating at open source events for Docker, Nodejs, etc... Developer Hackathons Paris and San Francisco in 2015 & 2016 Co-sponsoring with partners focused on a specific project goal (e.g., IP Drives, S3 API) Great for building visibility & community participation Next hackathon to develop creative extensions to Zenko! 42 Silicon Valley (free coding university) August in Fremont, CA: 22

24 Demo Zenko Installation & Portal 23

25 Release Plans Open Source Community Edition July 11th Available through github and docker Common API through S3 API Dual Server HA configuration (non scale-out) Backend store as volumes, AWS S3 Search Data Management Engine Location Control Enterprise Edition (EE) target beginning 2018 Scale-out solution Enterprise support 24 Search Engine Clueso Metadata Search Backbeat Data Workflow and MS Azure File BACKBEAT Management UI Open Source Community Edition September S3

26 Zenko EE: Enterprise Security, File & Scale-Out APP Enterprise Apps S3 CALLS S3 CALLS S3 API S3 API LOCATION LOCATION Scale Out NFS/ / SMB S3 API NFS/SMB LOCATION LOCATION Identity & Access Management (IAM): SAML 2.0/SSO with AD/LDAP DATA METADATA CRR/DATA METADATA CLUESO Metadata Search DATA STORAGE REST/Sproxyd AWS S3 API Scale-Out N-Way scale-out to any number of servers to deliver capacity AND performance for massive workloads, leverages the Metadata engine cluster from S3 Connector File & S3 Shared Access bi-directional file & object sharing with NFS v4/v3 & SMB for legacy apps Enables full Scale-Out for all key Zenko Services: METADATA: HA/Consistency Cluster DMD Multi-tenancy & Enterprise Security Full IAM support of Multi Accounts, Users, Groups, Policies & Single-Sign On (SSO) to AD & LDAP security servers DATA BACKBEAT Data Policy Engine Scale Out Zenko Enterprise Edition Legacy App APP AZURE BLOB API Google CS API Shared Local Storage Native Cloud Storage Support for multiple public clouds and Scality RING in native data format Backbeat for data workflow Policy-based data management engine Clueso for metadata search Apache Spark-based metadata search tool for optimal data insight

27 Getting involved with Zenko 26

28 How can I get involved with Zenko? Let us know what you do with Zenko stack! zenko@scality.com Get your project/company featured on the website in a quote Contribute tutorials Get a blogpost featuring your introduction of your tutorial Become part of our readthedocs hosted documentation Contribute code It s an opportunity to drive the roadmap with us! Join the team and be part of the Zenko craze! We have Contributing Guidelines on the GitHub repos, and we ll answer your questions via GitHub issues or our forum forum.scality.com Meet us at Microsoft Ignite, AWS Re:invent, Meetups... All info is on 27

29 Thank You

30 Thank You