TRANSFORMING CLOUD INFRASTRUCTURE TO SUPPORT BIG DATA STORAGE AND WORKFLOWS

Transcription

1 TRANSFORMING CLOUD INFRASTRUCTURE TO SUPPORT BIG DATA STORAGE AND WORKFLOWS

2 PRESENTER AND AGENDA PRESENTER Jay Migliaccio Director Cloud platforms and Services AGENDA Aspera Intro Big data and cloud storage challenges FASP protocol Deployment options / use cases Customer success Q&A

3 ASPERA S MISSION Creating next-generation transport technologies that move the world s digital assets at maximum speed, regardless of file size, transfer distance and network conditions. 3

4 TRENDS Big Data Explosion 90% of data today file-based or unstructured Mix of file sizes but larger and larger files the norm Diversity of IP Networks Media, Bandwidth Rates, and Conditions Variable bandwidth rates (slow to super-fast) Bandwidth rates increasing costs decreasing Network media remains diverse (terrestrial, satellite, wireless) Conditions vary all networks prone to degradation over distance Global Workflows moving Big Data over WANs Teams are geographically dispersed Over distance, network conditions degrade Contemporary TCP acceleration solutions not designed for big data transfer and replication Cloud Computing Grows Up Amazon Web Services (AWS) S3 cloud storage 2010: 262 billion objects, 2012: 1.3 trillion objects More choices: IBM SoftLayer, Microsoft Azure, OpenStack, HP Cloud No longer a niche Netflix (transcoding), MTV (global video distribution), BGI (genomic sequencing) 4

5 WHY CLOUD COMPUTING? THE POTENTIAL OF INFINITE COMPUTING RESOURCES, ON DEMAND Eliminates the need to plan ahead Allows companies to quickly meet changes in demand Without the lead-time bottleneck THE ELIMINATION OF AN UP-FRONT COMMITMENT Reduce capital outlay and investment risk Start small & increase hardware resources to match need Auto-scale to meet demand PAY-FOR-USE RESOURCE MODEL Virtual machines by the hour Storage by the month Bandwidth by the GB 5

6 BIG DATA CLOUD STORAGE CHALLENGE 6

7 CHALLENGES OF STORING BIG FILES IN THE CLOUD? DESIGNED AS SCALABLE DISTRIBUTED OBJECT STORES Google File System, Amazon S3, Hadoop Distributed File System Simple read/write of data blobs, indexed by a key Multiple replicas are distributed across storage for durability and optimized for high throughput access Should work well for storing large numbers of files BUT Blob" sizes are small (64 MB/128 MB) Large media files must be chunked 1TB file = transporting and reassembling 10,000+ chunks! Data I/O use the standard HTTP protocol VERY SLOW at distance Single HTTP stream slow even locally (<100 Mbps). Multi-chunk APIs impede workflow and are complex M&E/ BIG-DATA SERVICES REQUIRE HIGH-SPEED SOFTWARE BRIDGE OVER THE WAN Large files moved at full bandwidth capacity with global access Overcome the WAN and the I/O bottleneck Files of any size Transparent to the end user (GUI, command line, browser, etc.)

8 CHALLENGES FOR STORING BIG FILES IN DISTRIBUTED OBJECT CLOUD STORAGE APIs require Large files are divided into chunks (typically 64 MB MB) and multiple replicas are distributed across the storage for durability 1 TB file requires 10,000 chunks at 100MB per chunk! Optimized for high throughput access BUT only for *chunk* sizes in parallel, i.e. no high throughput, whole file read/write HTTP I/O protocol is HTTP only! HTTP PUT or GET by chunk SLOW over the WAN (due to TCP throughput bottleneck over WAN), e.g. <1-10s of Megabits/s depending on distance Even local I/O is slow unless a parallel HTTP stream write/read is used, e.g. local file system drivers, S3 fuse, are notoriously slow, e.g Megabytes/s Security and access control is only as good as the application Simply no tools for inter-cloud data transfer - Lock In! Most use cases need high speed transfer, virtually unlimited size, robust performance & security

9 BIG DATA CLOUD STORAGE CHALLENGE 9

10 CHALLENGES WITH TCP AND ALTERNATIVE TECHNOLOGIES Distance degrades conditions on all networks Latency (or Round Trip Times) increase Packet losses increase Fast networks just as prone to degradation TCP performance degrades with distance Throughput bottleneck becomes more severe with increased latency and packet loss TCP does not scale with bandwidth TCP designed for low bandwidth Adding more bandwidth does not improve throughput Alternative Technologies TCP-based - Network latency and packet loss must be low Modified TCP Improves TCP performance but insufficient for fast networks UDP traffic blasters - Inefficient and waste bandwidth Data caching - Inappropriate for many large file transfer workflows Data compression - Time consuming and impractical for certain file types CDNs & co-lo build outs - High overhead and expensive to scale 10

11 FASP HIGH-PERFORMANCE DATA TRANSPORT Maximum line-rate WAN transfer speed Transfer performance scales with bandwidth independent of transfer distance and resilient to packet loss Optimal end-to-end throughput efficiency Congestion Avoidance and Policy Control Automatic, full utilization of available bandwidth On-the-fly prioritization and bandwidth allocation Uncompromising security and reliability Secure, user/endpoint authentication AES-128 cryptography in transit & at-rest Scalable management, monitoring and control Real-time progress, performance and bandwidth utilization Detailed transfer history, logging, and manifest Enterprise-Class File Delivery Transfers up to thousands of times faster than FTP/HTTP(S) Precise and predictable transfer times Extreme scalability (concurrency and throughput) 11

12 FASP MANAGEMENT WITH ADAPTIVE RATE CONTROL Extraordinary bandwidth control Automatic, full utilization of available bandwidth Protection of other network traffic On-the-fly, per flow, user and job prioritization Highly-concurrent transfer stacking System-wide monitoring and reporting Real-time progress and performance analysis Real-time bandwidth utilization Detailed transfer history, logging and manifest Centralized command and control Per transfer, user, group and node Manage and create global transfer policies Remotely initiate, schedule and automate transfer jobs 12

13 BIG DATA CLOUD STORAGE CHALLENGE 13

14 PARTIAL SOLUTION MULTI-PART UPLOAD

15 Cloud ASPERA ON-DEMAND WITH DIRECT-TO-CLOUD

16 ASPERA TRANSFER PLATFORM ASPERA TRANSFER PLATFORM High-speed Transport Unified access to storage Security & Access Control Tracking and Reporting REST and SOAP APIs PARTNER ECOSYSTEM Automation Aspera Orchestrator Telestream Transformation Elemental encoding.com Sony CI theplatform Windows Azure Zencoder Media Management Harmonic Media Beacon Sony Vidispine Production Avid EVS Sharing / Collaboration Aspera Shares Aspera faspex 16

17 TRANSFERRING LARGE FILES WITH THE CLOUD FASP VS MULTIPART HTTP TRANSFER DATA TO CLOUD OVER WAN EFFECTIVE THROUGHPUT Typical internet conditions ms latency & 0.1 3% packet loss 15 parallel http streams <10 to 100 Mbps depending on distance Aspera FASP transfer over WAN to Cloud up to 1Gbps (per EC2 Extra Large Instance) independent of distance! 10 TB transferred per 24 hours 17

18 ASPERA ON DEMAND ADVANTAGES Maximum speed Enables large data set transfers over any network at maximum speed, regardless of network conditions or distance Transfers large data sets of small files with the same efficiency as large single files Very lightweight - does not require specialized or powerful hardware to maintain high speeds Adaptive rate control Provides precise rate control (pre-set and on-the-fly) for guaranteed transfer times Uses an adaptive rate control to fully utilize available bandwidth while remaining fair to other traffic Supports on-the-fly configurable bandwidth sharing policies. Users may pre-set and change individual transfer rates and finish times as needed Complete security Includes complete, built-in security using open standard cryptography for user authentication, data encryption and data integrity verification Robust, software only solution Uses standard, unmodified IP networking and is implemented in software as an application protocol.. Automatically resumes partial transfers and retries failed transfers Flexible open architecture Supports interoperable file and directory transfers between all major operating systems and provides a complete, modern software API to build upon

19 USE CASE: TRANSFER DIRECT-TO-OBJECT STORAGE 1. Upload using typical multi-part HTTP client 2. FASP high-speed upload Direct-to-Object storage Scale out HTTP multipart 2 FASP Transfer Server Aspera Client 1 HTTP multipart Cloud Platform Client, Emeryville, CA

20 USE CASE: INTRA-CLOUD TRANSFERS ACROSS SAME OR DIFFERENT CLOUD INFRASTRUCTURE THE SOLUTION Data migration from one region to another or from one provider to another Transfer database or application logs from one region to another for DR or Business Continuity FASP Node FASP Node FASP Node 20

21 USE CASE: SHARING AND COLLABORATION HYBRID ACROSS PUBLIC & PRIVATE CLOUDS THE SOLUTION Shares Web app transparently communicates with Aspera server Nodes and displays content in a single user interface User browses authorized content across multiple shares Independent high-speed data transfers to/from Datacenter, AWS S3, and Windows Azure BLOB, transparent to user FASP Client, NY FASP Shares DMZ Node Node Datacenter, Emeryville, CA Cloud Platform 21

22 USE CASE: PERSON-TO-PERSON DELIVERY WITH DIRECT-TO-OBJECT THE SOLUTION Faspex upload to cloud and notification to recipient Faspex download direct from cloud FASP FASP faspex Person-to- Person Direct to Object Connect Browser Plug-in Connect Browser Plug-in Los Angeles Cloud Platform New York 22

23 ASPERA TRANSFER SERVER INTEGRATED WITH CLOUD SERVICES HTTP 1 Client, NY FASP THE SOLUTION User authenticates to SaaS web application SaaS application communicates with Aspera via Node API Content is uploaded to and downloaded from Object storage via Aspera on Demand SaaS application accesses content in object storage for processing Aspera On Demand HTTP multipart Cloud Platform

24 ASPERA ON DEMAND: AUTO-SCALING PLATFORM KEY COMPONENTS Cluster Manager for Auto-scale and Scaled DB Console Management UI + Reporting API Enhanced Client for Shares Authorizations Unified Access to Files/Directories (Browser, GUI, Commend Line, SDK) AUTO SCALING Min/max number of t/s Utilization low/high watermark Min number of t/s in available pool Min number of idle t/s in available pool CLUSTER MGR Monitor cluster nodes Determine eligibility for transfer scale up / down Create / remove db with replicas Add / remove node

25 CUSTOMER SUCCESS STORIES Aspera On Demand

26 ULTIMATE FIGHTING CHAMPIONSHIP (UFC) About UFC UFC is the world s leading promoter of mixed martial arts (MMA), with programming broadcast to half a billion homes throughout 150 countries. Challenge Fights are located worldwide, from Ireland to Brazil to Las Vegas, Nevada, frequently in locations with poor connectivity. Transferring high-resolution video content from venue to host site in the cloud for scale-out transcoding Solution Aspera On Demand Application Platform running on AWS EC2 instances and Connect browser plug-in to transfer video clips at high-speed from the different venues to AWS. Production teams transfer the content from their laptops using the install-on-demand Connect plug-in, and the Aspera Application Platform stores the content directly in S3. Parallel encoding jobs are automatically started using encoding.com s cloud-based transcoding service, which also backstops on AWS, and the resulting output files are stored back into S3, ready for distribution. Final delivery of the device-specific content is accomplished using Amazon CloudFront, and all the URLs are automatically created and available for insertion into the UFC website and can be sent to 3 rd party content providers. 26

27 ULTIMATE FIGHTING CHAMPIONSHIP (UFC) 2 FASP HTTP multipart 1 Aspera Connect browser plug-in 4 Connect Server running on EC2 3 Parallel Transcoding 14 EC2 instances: 3 min Herndon, VA THE SOLUTION ` 1. Video Broadcast Capture 2. High-speed upload of content directly into AWS S3 3. Transfer completion triggers parallel transcoding in the cloud and device-specific videos are saved to S3 4. Using AWS CloudFront, content is delivered to viewers devices 27

28 NETFLIX TRANSCODING IN THE CLOUD About Netflix Netflix delivers streaming movies and TV shows to over 700 different devices PCs, internet-connected TVs, gaming consoles, tablets and smart phones. Challenge Scale-out transcoding without large upfront investments in compute and storage Fast turn around times for end-to-end processing, from content ingest to distribution Usage-based licensing to match timing of large content purchases Smooth transition for partners, minimal changes to their workflow Solution Aspera On Demand with Direct-to-S3 technology to move the content into the Amazon Web Services S3 storage for transcoding and streaming out to its customers. With Aspera On Demand Direct-to-S3, Netflix can achieve transfer rate improvements of up to 10x the typical transfer speeds of available cloud transfer solutions per virtual machine instance. Shares On Demand for content ingest to secure folders, abstracting the final location of the stored content 28

29 NETFLIX TRANSCODING IN THE CLOUD THE SOLUTION Shares Web app transparently communicates with Aspera server Nodes and displays content in a single user interface Aspera Client uploads directly via Aspera Transfer Servers CLI transfers also supported Independent high-speed data transfers to AWS S3 transparent to user FASP Browser Client FASP Shares Node Node Node Node Desktop Client

30 ROVIO ENTERTAINMENT About Rovio Entertainment Ltd is the creator of the globally successful Angry Birds which became an international phenomenon and has expanded rapidly into multifaceted entertainment, publishing and licensing to become a beloved international brand. Challenge Reliably move Angry Birds 52 part animation series to more than 20 broadcast partners around the world Solution Aspera faspex on Demand to accelerate content delivery to Rovio broadcast partners around the world and to streamline the related operational processes Each episode is first produced to high-quality animation, then encoded for each broadcaster Final output is delivered to broadcasters via Aspera faspex on Demand, avoiding the need for infrastructure build out and maintenance 30

31 THANK YOU Questions? FOR MORE INFORMATION ON ANY ASPERA PRODUCT, PLEASE VISIT