2013 AWS Worldwide Public Sector Summit Washington, D.C.

Size: px
Start display at page:

Download "2013 AWS Worldwide Public Sector Summit Washington, D.C."

Transcription

1 2013 AWS Worldwide Public Sector Summit Washington, D.C. EMR for Fun and for Profit Ben Butler Sr. Manager, Big

2 Overview 1. What is big data? 2. What is AWS Elastic MapReduce? 3. What data is available? 4. How to use AWS EMR to support my agency s mission? 2013 AWS Worldwide Public Sector Summit

3 What is Big Data?

4 Storage Big Data Compute So what is it? When your data sets become so large that you have to start innovating around how to collect, store, organize, analyze and share it

5 Storage Big Data Compute 100 GB Challenges start at relatively small volumes 1,000 PB

6 Storage Big Data Unconstrained data growth Compute EB ZB 95% of the 1.2 zettabytes of data in the digital universe is unstructured 70% of of this is usergenerated content GB TB PB Unstructured data growth explosive, with estimates of compound annual growth (CAGR) at 62% from Source: IDC

7 Storage Big Data Where does it come from? Compute Web sites Blogs/Reviews/ s/Pictures Social Graphs Facebook, Linked-in, Contacts Application server logs Web sites, games Sensor data Weather, water, smart grids Images/videos Traffic, security cameras Twitter 50m tweets/day 1,400% growth/year

8 Why AWS and big data? Storage Innovation Compute Amazon Amazon DynamoDB RedShift Amazon S3 HPC Spot EMR

9 Storage Big Data How do you get your slice of it? Compute AWS Direct Connect AWS Import/Export Queuing Amazon Storage Gateway Dedicated low latency bandwidth Physical media shipping Highly scalable event buffering Sync local storage to the cloud

10 Storage Big Data Where do you put your slice of it? Compute AWS Relational Database Service Fully managed database (MySQL, Oracle, MSSQL) AWS SimpleDB NoSQL, Schema-less Smaller datasets AWS DynamoDB NoSQL, Schema-less, Provisioned throughput database Amazon S3 Object datastore up to 5TB per object % durability

11 Storage Big Data Where do you put your slice of it? Compute Amazon Glacier Long term cold storage From $0.01 per GB/Month % durability

12 Storage Big Data How quick do you need to read it? Compute Single digit ms 10s-100s ms <5 hours AWS DynamoDB AWS S3 AWS Glacier Social scale applications Provisioned throughput performance Flexible consistency models Any object, any app % durability Objects up to 5TB in size Performance Media & asset archives Extremely low cost S3 levels of durability Scale Price

13 Storage Big Data Operate at any scale Compute Unlimited data Performance Scale Price

14 Storage Big Data Data has gravity Compute App Data App

15 Storage Big Data and inertia at volume Compute Data

16 Storage Big Data easier to move applications to the data Compute Data

17 Storage Big Data Bring compute capacity to the data Compute Very large dataset seeks strong & consistent compute for short term relationship, possibly longer

18 Storage Big Data Flexible compute resources, on demand Compute Amazon Elastic Compute Cloud (EC2) Basic unit of compute capacity Range of CPU, memory & local disk options 17 Instance types available, from micro through cluster compute to SSD backed Vertical Scaling From $0.02/hr Feature Flexible Scalable Machine Images Full control VM Import/Export Monitoring Inexpensive Secure Details Run Windows or Linux distributions Wide range of instance types from micro to cluster compute Configurations can be saved as machine images (AMIs) from which new instances can be created Full root or administrator rights Import and export VM images to transfer configurations in and out of EC2 Publishes metrics to Cloud Watch On-demand, Reserved and Spot instance types Full firewall control via Security Groups

19 Storage Big Data Elastic capacity as you need it Compute On and Off Fast Growth Variable peaks Predictable peaks

20 Storage Big Data Compute Elastic capacity as you need it WASTE On and Off Fast Growth Variable peaks CUSTOMER DISSATISFACTION Predictable peaks

21 Storage Big Data Elastic capacity as you need it Compute Capacity Traditional IT capacity Time Elastic cloud capacity Your IT needs

22 Storage Big Data Elastic capacity as you need it Compute On and Off Fast Growth Variable peaks Predictable peaks

23 Storage Big Data From one instance Compute

24 Storage Big Data to thousands Compute

25 Why AWS and big data? Storage Innovation Compute DynamoDB S3 RedShift HPC Spot EMR

26 Why AWS and big data? Storage Innovation Compute DynamoDB S3 RedShift HPC Spot EMR

27 AWS EMR Elastic MapReduce

28 Amazon Elastic MapReduce A key tool in the toolbox to help with Big Data challenges Makes possible analytics processes previously not feasible Cost effective when leveraged with EC2 spot market Broad ecosystem of tools to handle specific use cases 2013 AWS Worldwide Public Sector Summit

29 Hadoop-as-a-service Map-Reduce engine Integrated with tools What is EMR? Massively parallel Integrated to AWS services Cost effective AWS wrapper

30 Very large click log (e.g TBs)

31 Very large click log (e.g TBs) Lots of actions by John Smith

32 Lots of actions by John Smith Very large click log (e.g TBs) Split the log into many small pieces

33 Lots of actions by John Smith Process in an EMR cluster Very large click log (e.g TBs) Split the log into many small pieces

34 Lots of actions by John Smith Process in an EMR cluster Very large click log (e.g TBs) Split the log into many small pieces Aggregate the results from all the nodes

35 Lots of actions by John Smith Process in an EMR cluster Very large click log (e.g TBs) Split the log into many small pieces Aggregate the results from all the nodes What John Smith did

36 Very large click log (e.g TBs) Insight in a fraction of the time What John Smith did

37 How does it work? 1. Put the data into S3 (or HDFS) 3. Get the results Amazon S3 Amazon EMR Cluster EMR 2. Launch your cluster. Choose: Hadoop distribution How many nodes Node type (hi-cpu, hi-memory, etc.) Hadoop apps (Hive, Pig, HBase) 2013 AWS Worldwide Public Sector Summit

38 How does it work? Amazon EMR Cluster You can easily resize the cluster Amazon S3 EMR 2013 AWS Worldwide Public Sector Summit

39 How does it work? Amazon EMR Cluster Amazon S3 EMR Use Spot nodes to save time and money 2013 AWS Worldwide Public Sector Summit

40 How does it work? Amazon S3 Amazon EMR Clusters EMR Launch parallel clusters against the same data source (tune for the workload) 2013 AWS Worldwide Public Sector Summit

41 How does it work? Amazon S3 Amazon EMR Cluster When the work is complete, you can terminate the cluster (and stop paying) 2013 AWS Worldwide Public Sector Summit

42 How does it work? Amazon EMR Cluster You can store everything in HDFS (local disk) High Storage nodes = 48 TB/node 2013 AWS Worldwide Public Sector Summit

43 How does it work? Amazon EMR Cluster Launch in a Virtual Private Cloud for extra security 2013 AWS Worldwide Public Sector Summit

44 Start an Amazon EMR cluster using AWS Management Console or AWS Command Line Interface tools Amazon EMR cluster

45 Master instance group created that controls the cluster (runs MySQL) Amazon EMR cluster Master instance group

46 Core instance group created for life of cluster Amazon EMR cluster Master instance group Core instance group

47 Core instances run DataNode and TaskTracker daemons Amazon EMR cluster Master instance group HDFS HDFS Core instance group

48 Optional task instances can be added or subtracted to perform work Amazon EMR cluster Master instance group HDFS HDFS Core instance group Task instance group

49 Amazon S3 can be used as underlying file system for input/output data Amazon EMR cluster Master instance group HDFS HDFS Core instance group Task instance group Amazon S3

50 Master node coordinates distribution of work and manages cluster state Amazon EMR cluster Master instance group HDFS HDFS Core instance group Task instance group Amazon S3

51 Core and Task instances read-write to Amazon S3 Amazon EMR cluster Master instance group HDFS HDFS Core instance group Task instance group Amazon S3

52 Amazon EC2 instance Input file map Shuffle reduce Output file

53 Amazon EC2 instance Input file map reduce Output file EC2 instance Input file map reduce Output file EC2 instance Input file map Shuffle reduce Output file

54 Pig HDFS Amazon EMR

55 HDFS Amazon EMR Amazon S3 Amazon DynamoDB

56 Data management HDFS Amazon EMR Amazon S3 Amazon DynamoDB

57 Data management Analytics languages Pig HDFS Amazon EMR Amazon S3 Amazon DynamoDB

58 Data management Analytics languages Pig HDFS Amazon EMR Amazon RDS Amazon S3 Amazon DynamoDB

59 Data management Analytics languages Pig HDFS Amazon EMR Amazon RDS Amazon RedShift AWS Data Pipeline Amazon S3 Amazon DynamoDB

60 Amazon s Public Data Sets

61 Amazon Public Data Sets 2013 AWS Worldwide Public Sector Summit

62 1000 Genomes Project 270+ TB and growing dataset, hosted for free in AWS cloud Researchers no longer need massive onpremises storage and compute Collaboration revolution: not just shared data but executable papers 2013 AWS Worldwide Public Sector Summit

63 Using EMR for your Mission

64 Cloud Democratizes Big Data/HPC Lack of constraints leads to new usage models Gives control back to individual development teams Fail-fast (and fail-cheap) opens up exploratory style Many customers create 100s of Amazon EMR clusters per day Classic burst-y workload perfect for the cloud Big data / HPC clusters themselves are parallelized resources Can you build a faster on-premises cluster? Yes, but Usually a shared/contented resource; in cloud, each user/workgroup gets their own cluster Cloud is often the fastest platform based on MTTJC (Mean Time To Job Completion)

65 Cloud and Big Data/HPC No-obligation use allows for experimentation, prototypes and operational/business pilots Faster time from inception of idea to solution Provides a platform that can scale to meet the massive needs of large data sets Bottom line: Enables experimentation and innovation without large capital investments Improves ROI for Big Data projects

66 Try the tutorial: aws.amazon.com/articles/2855 Find out more: aws.amazon.com/big-data

67 Thank you! Ben Butler, Sr. Manager, AWS Marketing

Cloud Analytics and Business Intelligence on AWS

Cloud Analytics and Business Intelligence on AWS Cloud Analytics and Business Intelligence on AWS Enterprise Applications Virtual Desktops Sharing & Collaboration Platform Services Analytics Hadoop Real-time Streaming Data Machine Learning Data Warehouse

More information

Big Data on AWS. Big Data Agility and Performance Delivered in the Cloud. 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Big Data on AWS. Big Data Agility and Performance Delivered in the Cloud. 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Big Data on AWS Big Data Agility and Performance Delivered in the Cloud 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Big Data Technologies and techniques for working productively

More information

CIT 668: System Architecture. Amazon Web Services

CIT 668: System Architecture. Amazon Web Services CIT 668: System Architecture Amazon Web Services Topics 1. AWS Global Infrastructure 2. Foundation Services 1. Compute 2. Storage 3. Database 4. Network 3. AWS Economics Amazon Services Architecture Regions

More information

What is Cloud Computing? What are the Private and Public Clouds? What are IaaS, PaaS, and SaaS? What is the Amazon Web Services (AWS)?

What is Cloud Computing? What are the Private and Public Clouds? What are IaaS, PaaS, and SaaS? What is the Amazon Web Services (AWS)? What is Cloud Computing? What are the Private and Public Clouds? What are IaaS, PaaS, and SaaS? What is the Amazon Web Services (AWS)? What is Amazon Machine Image (AMI)? Amazon Elastic Compute Cloud (EC2)?

More information

Big Data on AWS. Peter-Mark Verwoerd Solutions Architect

Big Data on AWS. Peter-Mark Verwoerd Solutions Architect Big Data on AWS Peter-Mark Verwoerd Solutions Architect What to get out of this talk Non-technical: Big Data processing stages: ingest, store, process, visualize Hot vs. Cold data Low latency processing

More information

AWS: Basic Architecture Session SUNEY SHARMA Solutions Architect: AWS

AWS: Basic Architecture Session SUNEY SHARMA Solutions Architect: AWS AWS: Basic Architecture Session SUNEY SHARMA Solutions Architect: AWS suneys@amazon.com AWS Core Infrastructure and Services Traditional Infrastructure Amazon Web Services Security Security Firewalls ACLs

More information

Agenda. AWS Database Services Traditional vs AWS Data services model Amazon RDS Redshift DynamoDB ElastiCache

Agenda. AWS Database Services Traditional vs AWS Data services model Amazon RDS Redshift DynamoDB ElastiCache Databases on AWS 2017 Amazon Web Services, Inc. and its affiliates. All rights served. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon Web Services,

More information

Best Practices and Performance Tuning on Amazon Elastic MapReduce

Best Practices and Performance Tuning on Amazon Elastic MapReduce Best Practices and Performance Tuning on Amazon Elastic MapReduce Michael Hanisch Solutions Architect Amo Abeyaratne Big Data and Analytics Consultant ANZ 12.04.2016 2016, Amazon Web Services, Inc. or

More information

AWS Solutions Architect Associate (SAA-C01) Sample Exam Questions

AWS Solutions Architect Associate (SAA-C01) Sample Exam Questions 1) A company is storing an access key (access key ID and secret access key) in a text file on a custom AMI. The company uses the access key to access DynamoDB tables from instances created from the AMI.

More information

Amazon AWS-Solution-Architect-Associate Exam

Amazon AWS-Solution-Architect-Associate Exam Volume: 858 Questions Question: 1 You are trying to launch an EC2 instance, however the instance seems to go into a terminated status immediately. What would probably not be a reason that this is happening?

More information

Cloud Computing 1. CSCI 4850/5850 High-Performance Computing Spring 2018

Cloud Computing 1. CSCI 4850/5850 High-Performance Computing Spring 2018 Cloud Computing 1 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning

More information

Introduction to Amazon Web Services. Jeff Barr Senior AWS /

Introduction to Amazon Web Services. Jeff Barr Senior AWS / Introduction to Amazon Web Services Jeff Barr Senior AWS Evangelist @jeffbarr / jbarr@amazon.com What Does It Take to be a Global Online Retailer? The Obvious Part And the Not-So Obvious Part How Did

More information

8/3/17. Encryption and Decryption centralized Single point of contact First line of defense. Bishop

8/3/17. Encryption and Decryption centralized Single point of contact First line of defense. Bishop Bishop Encryption and Decryption centralized Single point of contact First line of defense If working with VPC Creation and management of security groups Provides additional networking and security options

More information

AWS Storage Optimization. AWS Whitepaper

AWS Storage Optimization. AWS Whitepaper AWS Storage Optimization AWS Whitepaper AWS Storage Optimization: AWS Whitepaper Copyright 2018 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress

More information

Introducing Amazon Elastic File System (EFS)

Introducing Amazon Elastic File System (EFS) Introducing Amazon Elastic File System (EFS) Danilo Poccia, Technical Evangelist, AWS @danilop 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Goals and expectations for this session

More information

What s New at AWS? A selection of some new stuff. Constantin Gonzalez, Principal Solutions Architect, Amazon Web Services

What s New at AWS? A selection of some new stuff. Constantin Gonzalez, Principal Solutions Architect, Amazon Web Services What s New at AWS? A selection of some new stuff Constantin Gonzalez, Principal Solutions Architect, Amazon Web Services Speed of Innovation AWS Pace of Innovation AWS has been continually expanding its

More information

Werden Sie ein Teil von Internet der Dinge auf AWS. AWS Enterprise Summit 2015 Dr. Markus Schmidberger -

Werden Sie ein Teil von Internet der Dinge auf AWS. AWS Enterprise Summit 2015 Dr. Markus Schmidberger - Werden Sie ein Teil von Internet der Dinge auf AWS AWS Enterprise Summit 2015 Dr. Markus Schmidberger - schmidbe@amazon.de Internet of Things is the network of physical objects or "things" embedded with

More information

CS15-319: Cloud Computing. Lecture 3 Course Project and Amazon AWS Majd Sakr and Mohammad Hammoud

CS15-319: Cloud Computing. Lecture 3 Course Project and Amazon AWS Majd Sakr and Mohammad Hammoud CS15-319: Cloud Computing Lecture 3 Course Project and Amazon AWS Majd Sakr and Mohammad Hammoud Lecture Outline Discussion On Course Project Amazon Web Services 2 Course Project Course Project Phase I-A

More information

QLIK INTEGRATION WITH AMAZON REDSHIFT

QLIK INTEGRATION WITH AMAZON REDSHIFT QLIK INTEGRATION WITH AMAZON REDSHIFT Qlik Partner Engineering Created August 2016, last updated March 2017 Contents Introduction... 2 About Amazon Web Services (AWS)... 2 About Amazon Redshift... 2 Qlik

More information

Introduction to Database Services

Introduction to Database Services Introduction to Database Services Shaun Pearce AWS Solutions Architect 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Today s agenda Why managed database services? A non-relational

More information

Amazon Web Services. For Government, Education, and Nonprofit Organizations

Amazon Web Services. For Government, Education, and Nonprofit Organizations Amazon Web Services For Government, Education, and Nonprofit Organizations Max Peterson GM EMEA, LATAM and Global Contracts maxpete@amazon.co.uk +44 (0)7342 079563 2015, Amazon Web Services, Inc. or its

More information

Building High Performance Apps using NoSQL. Swami Sivasubramanian General Manager, AWS NoSQL

Building High Performance Apps using NoSQL. Swami Sivasubramanian General Manager, AWS NoSQL Building High Performance Apps using NoSQL Swami Sivasubramanian General Manager, AWS NoSQL Building high performance apps There is a lot to building high performance apps Scalability Performance at high

More information

AWS Storage Gateway. Not your father s hybrid storage. University of Arizona IT Summit October 23, Jay Vagalatos, AWS Solutions Architect

AWS Storage Gateway. Not your father s hybrid storage. University of Arizona IT Summit October 23, Jay Vagalatos, AWS Solutions Architect AWS Storage Gateway Not your father s hybrid storage University of Arizona IT Summit 2017 Jay Vagalatos, AWS Solutions Architect October 23, 2017 The AWS Storage Portfolio Amazon EBS (persistent) Block

More information

About Intellipaat. About the Course. Why Take This Course?

About Intellipaat. About the Course. Why Take This Course? About Intellipaat Intellipaat is a fast growing professional training provider that is offering training in over 150 most sought-after tools and technologies. We have a learner base of 600,000 in over

More information

Cloud Storage with AWS: EFS vs EBS vs S3 AHMAD KARAWASH

Cloud Storage with AWS: EFS vs EBS vs S3 AHMAD KARAWASH Cloud Storage with AWS: EFS vs EBS vs S3 AHMAD KARAWASH Cloud Storage with AWS Cloud storage is a critical component of cloud computing, holding the information used by applications. Big data analytics,

More information

Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect

Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect Igor Roiter Big Data Cloud Solution Architect Working as a Data Specialist for the last 11 years 9 of them as a Consultant specializing

More information

Embedded Technosolutions

Embedded Technosolutions Hadoop Big Data An Important technology in IT Sector Hadoop - Big Data Oerie 90% of the worlds data was generated in the last few years. Due to the advent of new technologies, devices, and communication

More information

Introduction: Is Amazon Web Service (AWS) cloud supports best cost effective & high performance modern disaster recovery.

Introduction: Is Amazon Web Service (AWS) cloud supports best cost effective & high performance modern disaster recovery. Is Amazon Web Service (AWS) cloud supports best cost effective & high performance modern disaster recovery. Name: Rajib Das Employee ID- tcs 231462 ISU-CSP Email-rajib2.d@tcs.com Introduction: Disaster

More information

Cloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018

Cloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018 Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning

More information

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?

More information

BUSINESS DATA LAKE FADI FAKHOURI, SR. SYSTEMS ENGINEER, ISILON SPECIALIST. Copyright 2016 EMC Corporation. All rights reserved.

BUSINESS DATA LAKE FADI FAKHOURI, SR. SYSTEMS ENGINEER, ISILON SPECIALIST. Copyright 2016 EMC Corporation. All rights reserved. BUSINESS DATA LAKE FADI FAKHOURI, SR. SYSTEMS ENGINEER, ISILON SPECIALIST 1 UNSTRUCTURED DATA GROWTH 75% 78% 80% 2015 71 EB 2016 106 EB 2017 133 EB Total Capacity Shipped, Worldwide % of Unstructured Data

More information

Next Generation Storage for The Software-Defned World

Next Generation Storage for The Software-Defned World ` Next Generation Storage for The Software-Defned World John Hofer Solution Architect Red Hat, Inc. BUSINESS PAINS DEMAND NEW MODELS CLOUD ARCHITECTURES PROPRIETARY/TRADITIONAL ARCHITECTURES High up-front

More information

What s New at AWS? looking at just a few new things for Enterprise. Philipp Behre, Enterprise Solutions Architect, Amazon Web Services

What s New at AWS? looking at just a few new things for Enterprise. Philipp Behre, Enterprise Solutions Architect, Amazon Web Services What s New at AWS? looking at just a few new things for Enterprise Philipp Behre, Enterprise Solutions Architect, Amazon Web Services 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

More information

Tour of Database Platforms as a Service. June 2016 Warner Chaves Christo Kutrovsky Solutions Architect

Tour of Database Platforms as a Service. June 2016 Warner Chaves Christo Kutrovsky Solutions Architect Tour of Database Platforms as a Service June 2016 Warner Chaves Christo Kutrovsky Solutions Architect Bio Solutions Architect at Pythian Specialize high performance data processing and analytics 15 years

More information

CLOUD ECONOMICS: HOW TO QUANTIFY THE BENEFITS OF MOVING TO THE CLOUD

CLOUD ECONOMICS: HOW TO QUANTIFY THE BENEFITS OF MOVING TO THE CLOUD CLOUD ECONOMICS: HOW TO QUANTIFY THE BENEFITS OF MOVING TO THE CLOUD Matt Johnson, Solutions Architect, Amazon Web Services 30 October 2017 2017, Amazon Web Services, Inc. or its Affiliates. All rights

More information

AWS & Intel: A Partnership Dedicated to fueling your Innovations. Thomas Kellerer BDM CSP, Intel Central Europe

AWS & Intel: A Partnership Dedicated to fueling your Innovations. Thomas Kellerer BDM CSP, Intel Central Europe AWS & Intel: A Partnership Dedicated to fueling your Innovations Thomas Kellerer BDM CSP, Intel Central Europe The Digital Service Economy Growth in connected devices enables new business opportunities

More information

ThoughtSpot on AWS Quick Start Guide

ThoughtSpot on AWS Quick Start Guide ThoughtSpot on AWS Quick Start Guide Version 4.2 February 2017 Table of Contents Contents Chapter 1: Welcome to ThoughtSpot...3 Contact ThoughtSpot... 4 Chapter 2: Introduction... 6 About AWS...7 Chapter

More information

Amazon Web Services. Foundational Services for Research Computing. April Mike Kuentz, WWPS Solutions Architect

Amazon Web Services. Foundational Services for Research Computing. April Mike Kuentz, WWPS Solutions Architect Amazon Web Services Foundational Services for Research Computing Mike Kuentz, WWPS Solutions Architect April 2017 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Global Infrastructure

More information

Energy Management with AWS

Energy Management with AWS Energy Management with AWS Kyle Hart and Nandakumar Sreenivasan Amazon Web Services August [XX], 2017 Tampa Convention Center Tampa, Florida What is Cloud? The NIST Definition Broad Network Access On-Demand

More information

Project Presentation

Project Presentation Project Presentation Saad Arif Dept. of Electrical Engineering and Computer Science University of Central Florida - Orlando, FL November 7, 2013 1 Introduction 1 Introduction 2 Gallery 1 Introduction 2

More information

Big Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ.

Big Data Programming: an Introduction. Spring 2015, X. Zhang Fordham Univ. Big Data Programming: an Introduction Spring 2015, X. Zhang Fordham Univ. Outline What the course is about? scope Introduction to big data programming Opportunity and challenge of big data Origin of Hadoop

More information

Spatial Analytics Built for Big Data Platforms

Spatial Analytics Built for Big Data Platforms Spatial Analytics Built for Big Platforms Roberto Infante Software Development Manager, Spatial and Graph 1 Copyright 2011, Oracle and/or its affiliates. All rights Global Digital Growth The Internet of

More information

Deep Dive on Amazon Elastic File System

Deep Dive on Amazon Elastic File System Deep Dive on Amazon Elastic File System Yong S. Kim AWS Business Development Manager, Amazon EFS Paul Moran Technical Account Manager, Enterprise Support 28 th of June 2017 2015, Amazon Web Services, Inc.

More information

AUTOMATING IBM SPECTRUM SCALE CLUSTER BUILDS IN AWS PROOF OF CONCEPT

AUTOMATING IBM SPECTRUM SCALE CLUSTER BUILDS IN AWS PROOF OF CONCEPT AUTOMATING IBM SPECTRUM SCALE CLUSTER BUILDS IN AWS PROOF OF CONCEPT By Joshua Kwedar Sr. Systems Engineer By Steve Horan Cloud Architect ATS Innovation Center, Malvern, PA Dates: Oct December 2017 INTRODUCTION

More information

Amazon Web Services 101 April 17 th, 2014 Joel Williams Solutions Architect. Amazon.com, Inc. and its affiliates. All rights reserved.

Amazon Web Services 101 April 17 th, 2014 Joel Williams Solutions Architect. Amazon.com, Inc. and its affiliates. All rights reserved. Amazon Web Services 101 April 17 th, 2014 Joel Williams Solutions Architect Amazon.com, Inc. and its affiliates. All rights reserved. Learning about Cloud Computing with AWS What is Cloud Computing and

More information

Amazon Linux: Operating System of the Cloud

Amazon Linux: Operating System of the Cloud Amazon Linux: Operating System of the Cloud Chris Schlaeger Director, Kernel and Operating Systems Managing Director, Amazon Development Center Germany GmbH How did Amazon get into Cloud Computing? We

More information

Leveraging Software-Defined Storage to Meet Today and Tomorrow s Infrastructure Demands

Leveraging Software-Defined Storage to Meet Today and Tomorrow s Infrastructure Demands Leveraging Software-Defined Storage to Meet Today and Tomorrow s Infrastructure Demands Unleash Your Data Center s Hidden Power September 16, 2014 Molly Rector CMO, EVP Product Management & WW Marketing

More information

Accelerate Big Data Insights

Accelerate Big Data Insights Accelerate Big Data Insights Executive Summary An abundance of information isn t always helpful when time is of the essence. In the world of big data, the ability to accelerate time-to-insight can not

More information

Middle East Technical University. Jeren AKHOUNDI ( ) Ipek Deniz Demirtel ( ) Derya Nur Ulus ( ) CENG553 Database Management Systems

Middle East Technical University. Jeren AKHOUNDI ( ) Ipek Deniz Demirtel ( ) Derya Nur Ulus ( ) CENG553 Database Management Systems Middle East Technical University Jeren AKHOUNDI (1836345) Ipek Deniz Demirtel (1997691) Derya Nur Ulus (1899608) CENG553 Database Management Systems * Introduction to Cloud Computing * Cloud DataBase as

More information

Automating Elasticity. March 2018

Automating Elasticity. March 2018 Automating Elasticity March 2018 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document is provided for informational purposes only. It represents AWS s current product

More information

EMC STORAGE STRATEGY. Copyright 2015 EMC Corporation. All rights reserved.

EMC STORAGE STRATEGY. Copyright 2015 EMC Corporation. All rights reserved. EMC STORAGE STRATEGY 1 CREATING A MAJOR PAIN POINT FOR CIO S(BI-MODAL IT) MANAGE RISK INVEST REDUCE COST WORKLOAD CHARACTERISTICS Performance Traditional Apps Sequential Stability Reliability Greater Impact

More information

SoftNAS Cloud Data Management Products for AWS Add Breakthrough NAS Performance, Protection, Flexibility

SoftNAS Cloud Data Management Products for AWS Add Breakthrough NAS Performance, Protection, Flexibility Control Any Data. Any Cloud. Anywhere. SoftNAS Cloud Data Management Products for AWS Add Breakthrough NAS Performance, Protection, Flexibility Understanding SoftNAS Cloud SoftNAS, Inc. is the #1 software-defined

More information

White Paper. Platform9 ROI for Hybrid Clouds

White Paper. Platform9 ROI for Hybrid Clouds White Paper Platform9 ROI for Hybrid Clouds Quantifying the cost savings and benefits of moving from the Amazon Web Services (AWS) public cloud to the Platform9 hybrid cloud. Abstract Deciding whether

More information

DATABASE SCALE WITHOUT LIMITS ON AWS

DATABASE SCALE WITHOUT LIMITS ON AWS The move to cloud computing is changing the face of the computer industry, and at the heart of this change is elastic computing. Modern applications now have diverse and demanding requirements that leverage

More information

Benefits of Extending your Datacenters with Amazon Web Services

Benefits of Extending your Datacenters with Amazon Web Services Benefits of Extending your Datacenters with Amazon Web Services Xavier Prélat Business Development Manager @aws_actus How did Amazon.. get into cloud computing? What is AWS? Amazon Web Services offers

More information

MixApart: Decoupled Analytics for Shared Storage Systems. Madalin Mihailescu, Gokul Soundararajan, Cristiana Amza University of Toronto and NetApp

MixApart: Decoupled Analytics for Shared Storage Systems. Madalin Mihailescu, Gokul Soundararajan, Cristiana Amza University of Toronto and NetApp MixApart: Decoupled Analytics for Shared Storage Systems Madalin Mihailescu, Gokul Soundararajan, Cristiana Amza University of Toronto and NetApp Hadoop Pig, Hive Hadoop + Enterprise storage?! Shared storage

More information

Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015

Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015 Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL May 2015 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document

More information

Cloud Computing & Visualization

Cloud Computing & Visualization Cloud Computing & Visualization Workflows Distributed Computation with Spark Data Warehousing with Redshift Visualization with Tableau #FIUSCIS School of Computing & Information Sciences, Florida International

More information

How can you implement this through a script that a scheduling daemon runs daily on the application servers?

How can you implement this through a script that a scheduling daemon runs daily on the application servers? You ve been tasked with implementing an automated data backup solution for your application servers that run on Amazon EC2 with Amazon EBS volumes. You want to use a distributed data store for your backups

More information

Backup & Recovery on AWS

Backup & Recovery on AWS EBOOK: Backup & Recovery on AWS 2017, Amazon Web Services, Inc. or its affiliates. All rights reserved. Contents Backup and Recovery on AWS AWS Object Storage Services RDX APN Storage Partner Benefits

More information

Integrating Splunk with AWS services:

Integrating Splunk with AWS services: Integrating Splunk with AWS services: Using Redshi+, Elas0c Map Reduce (EMR), Amazon Machine Learning & S3 to gain ac0onable insights via predic0ve analy0cs via Splunk Patrick Shumate Solutions Architect,

More information

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Dr. Steffen Hausmann, Solutions Architect Michael Hanisch, Manager Solutions Architecture November 18 th, 2016 Stream Processing Challenges

More information

What is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed?

What is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed? Simple to start What is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed? What is the maximum download speed you get? Simple computation

More information

Lean & Mean on AWS: Cost-Effective Architectures. Constantin Gonzalez, Solutions Architect, AWS

Lean & Mean on AWS: Cost-Effective Architectures. Constantin Gonzalez, Solutions Architect, AWS Lean & Mean on AWS: Cost-Effective Architectures Constantin Gonzalez, Solutions Architect, AWS What you ll get out of this session A lower AWS bill A more scalable, robust, dynamic architecture More time

More information

CISC 7610 Lecture 2b The beginnings of NoSQL

CISC 7610 Lecture 2b The beginnings of NoSQL CISC 7610 Lecture 2b The beginnings of NoSQL Topics: Big Data Google s infrastructure Hadoop: open google infrastructure Scaling through sharding CAP theorem Amazon s Dynamo 5 V s of big data Everyone

More information

Amazon Web Services Training. Training Topics:

Amazon Web Services Training. Training Topics: Amazon Web Services Training Training Topics: SECTION1: INTRODUCTION TO CLOUD COMPUTING A Short history Client Server Computing Concepts Challenges with Distributed Computing Introduction to Cloud Computing

More information

Big Data with Hadoop Ecosystem

Big Data with Hadoop Ecosystem Diógenes Pires Big Data with Hadoop Ecosystem Hands-on (HBase, MySql and Hive + Power BI) Internet Live http://www.internetlivestats.com/ Introduction Business Intelligence Business Intelligence Process

More information

STATE OF MODERN APPLICATIONS IN THE CLOUD

STATE OF MODERN APPLICATIONS IN THE CLOUD STATE OF MODERN APPLICATIONS IN THE CLOUD 2017 Introduction The Rise of Modern Applications What is the Modern Application? Today s leading enterprises are striving to deliver high performance, highly

More information

AWS 101. Patrick Pierson, IonChannel

AWS 101. Patrick Pierson, IonChannel AWS 101 Patrick Pierson, IonChannel What is AWS? Amazon Web Services (AWS) is a secure cloud services platform, offering compute power, database storage, content delivery and other functionality to help

More information

Next-Generation Cloud Platform

Next-Generation Cloud Platform Next-Generation Cloud Platform Jangwoo Kim Jun 24, 2013 E-mail: jangwoo@postech.ac.kr High Performance Computing Lab Department of Computer Science & Engineering Pohang University of Science and Technology

More information

Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018

Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018 Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster

More information

Advanced Architectures for Oracle Database on Amazon EC2

Advanced Architectures for Oracle Database on Amazon EC2 Advanced Architectures for Oracle Database on Amazon EC2 Abdul Sathar Sait Jinyoung Jung Amazon Web Services November 2014 Last update: April 2016 Contents Abstract 2 Introduction 3 Oracle Database Editions

More information

CSE6331: Cloud Computing

CSE6331: Cloud Computing CSE6331: Cloud Computing Leonidas Fegaras University of Texas at Arlington c 2019 by Leonidas Fegaras Cloud Computing Fundamentals Based on: J. Freire s class notes on Big Data http://vgc.poly.edu/~juliana/courses/bigdata2016/

More information

IBM Power Systems: Open innovation to put data to work Dexter Henderson Vice President IBM Power Systems

IBM Power Systems: Open innovation to put data to work Dexter Henderson Vice President IBM Power Systems IBM Power Systems: Open innovation to put data to work Dexter Henderson Vice President IBM Power Systems 2014 IBM Corporation Powerful Forces are Changing the Way Business Gets Done Data growing exponentially

More information

AWS Storage Gateway. Amazon S3. Amazon EFS. Amazon Glacier. Amazon EBS. Amazon EC2 Instance. storage. File Block Object. Hybrid integrated.

AWS Storage Gateway. Amazon S3. Amazon EFS. Amazon Glacier. Amazon EBS. Amazon EC2 Instance. storage. File Block Object. Hybrid integrated. AWS Storage Amazon EFS Amazon EBS Amazon EC2 Instance storage Amazon S3 Amazon Glacier AWS Storage Gateway File Block Object Hybrid integrated storage Amazon S3 Amazon Glacier Amazon EBS Amazon EFS Durable

More information

AWS Database Migration Service

AWS Database Migration Service AWS Database Migration Service Database Modernisation with Minimal Downtime John Winford Sr. Technical Program Manager May 18, 2017 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

More information

Introduction to Amazon Cloud & EC2 Overview

Introduction to Amazon Cloud & EC2 Overview Introduction to Amazon Cloud & EC2 Overview 2015 Amazon Web Services, Inc. and its affiliates. All rights served. May not be copied, modified, or distributed in whole or in part without the express consent

More information

Technical Brief. Adding Zadara Storage to VMware Cloud on AWS

Technical Brief. Adding Zadara Storage to VMware Cloud on AWS Technical Brief Adding Zadara Storage to VMware Cloud on AWS Revision History Row Version Date Description 1 1.0 26 Nov 2018 Initial release 2 1.1 15 Jan 2019 Updates for VMware style guide compliance,

More information

Pass4test Certification IT garanti, The Easy Way!

Pass4test Certification IT garanti, The Easy Way! Pass4test Certification IT garanti, The Easy Way! http://www.pass4test.fr Service de mise à jour gratuit pendant un an Exam : SOA-C01 Title : AWS Certified SysOps Administrator - Associate Vendor : Amazon

More information

Demystifying the Cloud With a Look at Hybrid Hosting and OpenStack

Demystifying the Cloud With a Look at Hybrid Hosting and OpenStack Demystifying the Cloud With a Look at Hybrid Hosting and OpenStack Robert Collazo Systems Engineer Rackspace Hosting The Rackspace Vision Agenda Truly a New Era of Computing 70 s 80 s Mainframe Era 90

More information

Amazon Elastic File System

Amazon Elastic File System Amazon Elastic File System Choosing Between the Different Throughput & Performance Modes July 2018 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document is provided

More information

Amazon Web Services (AWS) Solutions Architect Intermediate Level Course Content

Amazon Web Services (AWS) Solutions Architect Intermediate Level Course Content Amazon Web Services (AWS) Solutions Architect Intermediate Level Course Content Introduction to Cloud Computing A Short history Client Server Computing Concepts Challenges with Distributed Computing Introduction

More information

EBOOK: VMware Cloud on AWS: Optimized for the Next-Generation Hybrid Cloud

EBOOK: VMware Cloud on AWS: Optimized for the Next-Generation Hybrid Cloud EBOOK: VMware Cloud on AWS: Optimized for the Next-Generation Hybrid Cloud Contents Introduction... 3 What is VMware Cloud on AWS?... 5 Customer Benefits of Adopting VMware Cloud on AWS... 6 VMware Cloud

More information

Introduction to Cloud Computing

Introduction to Cloud Computing You will learn how to: Build and deploy cloud applications and develop an effective implementation strategy Leverage cloud vendors Amazon EC2 and Amazon S3 Exploit Software as a Service (SaaS) to optimize

More information

IBM Spectrum NAS, IBM Spectrum Scale and IBM Cloud Object Storage

IBM Spectrum NAS, IBM Spectrum Scale and IBM Cloud Object Storage IBM Spectrum NAS, IBM Spectrum Scale and IBM Cloud Object Storage Silverton Consulting, Inc. StorInt Briefing 2017 SILVERTON CONSULTING, INC. ALL RIGHTS RESERVED Page 2 Introduction Unstructured data has

More information

Bringing OpenStack to the Enterprise. An enterprise-class solution ensures you get the required performance, reliability, and security

Bringing OpenStack to the Enterprise. An enterprise-class solution ensures you get the required performance, reliability, and security Bringing OpenStack to the Enterprise An enterprise-class solution ensures you get the required performance, reliability, and security INTRODUCTION Organizations today frequently need to quickly get systems

More information

AWS Agility + Splunk Visibility = Cloud Success. Splunk App for AWS Demo. Laura Ripans, AWS Alliance Manager

AWS Agility + Splunk Visibility = Cloud Success. Splunk App for AWS Demo. Laura Ripans, AWS Alliance Manager AWS Agility + Splunk Visibility = Cloud Success Splunk App for AWS Demo Laura Ripans, AWS Alliance Manager Disruptive innovation and business transformation starts with data I HAVE BEEN GIVEN AN AWS ACCOUNT!!!

More information

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Big Data Technology Ecosystem Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Agenda End-to-End Data Delivery Platform Ecosystem of Data Technologies Mapping an End-to-End Solution Case

More information

CPET 581 Cloud Computing: Technologies and Enterprise IT Strategies

CPET 581 Cloud Computing: Technologies and Enterprise IT Strategies CPET 581 Cloud Computing: Technologies and Enterprise IT Strategies Lecture 8 Cloud Programming & Software Environments: High Performance Computing & AWS Services Part 2 of 2 Spring 2015 A Specialty Course

More information

Managing IoT and Time Series Data with Amazon ElastiCache for Redis

Managing IoT and Time Series Data with Amazon ElastiCache for Redis Managing IoT and Time Series Data with ElastiCache for Redis Darin Briskman, ElastiCache Developer Outreach Michael Labib, Specialist Solutions Architect 2016, Web Services, Inc. or its Affiliates. All

More information

Hadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved

Hadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved Hadoop 2.x Core: YARN, Tez, and Spark YARN Hadoop Machine Types top-of-rack switches core switch client machines have client-side software used to access a cluster to process data master nodes run Hadoop

More information

Cloud Computing 4/17/2016. Outline. Cloud Computing. Centralized versus Distributed Computing Some people argue that Cloud Computing. Cloud Computing.

Cloud Computing 4/17/2016. Outline. Cloud Computing. Centralized versus Distributed Computing Some people argue that Cloud Computing. Cloud Computing. Cloud Computing By: Muhammad Naseem Assistant Professor Department of Computer Engineering, Sir Syed University of Engineering & Technology, Web: http://sites.google.com/site/muhammadnaseem105 Email: mnaseem105@yahoo.com

More information

Apache Hadoop Goes Realtime at Facebook. Himanshu Sharma

Apache Hadoop Goes Realtime at Facebook. Himanshu Sharma Apache Hadoop Goes Realtime at Facebook Guide - Dr. Sunny S. Chung Presented By- Anand K Singh Himanshu Sharma Index Problem with Current Stack Apache Hadoop and Hbase Zookeeper Applications of HBase at

More information

Security & Compliance in the AWS Cloud. Vijay Rangarajan Senior Cloud Architect, ASEAN Amazon Web

Security & Compliance in the AWS Cloud. Vijay Rangarajan Senior Cloud Architect, ASEAN Amazon Web Security & Compliance in the AWS Cloud Vijay Rangarajan Senior Cloud Architect, ASEAN Amazon Web Services @awscloud www.cloudsec.com #CLOUDSEC Security & Compliance in the AWS Cloud TECHNICAL & BUSINESS

More information

Amazon Web Services (AWS) Training Course Content

Amazon Web Services (AWS) Training Course Content Amazon Web Services (AWS) Training Course Content SECTION 1: CLOUD COMPUTING INTRODUCTION History of Cloud Computing Concept of Client Server Computing Distributed Computing and it s Challenges What is

More information

PrepAwayExam. High-efficient Exam Materials are the best high pass-rate Exam Dumps

PrepAwayExam.   High-efficient Exam Materials are the best high pass-rate Exam Dumps PrepAwayExam http://www.prepawayexam.com/ High-efficient Exam Materials are the best high pass-rate Exam Dumps Exam : SAA-C01 Title : AWS Certified Solutions Architect - Associate (Released February 2018)

More information

TESTING BIG DATA WORLD RIGA. by Konstantin Pletenev OCTOBER, 2017, TAPOST GROW CONFIDENTLY

TESTING BIG DATA WORLD RIGA. by Konstantin Pletenev OCTOBER, 2017, TAPOST GROW CONFIDENTLY RIGA TESTING BIG DATA WORLD by Konstantin Pletenev OCTOBER, 2017, TAPOST GROW CONFIDENTLY BIG DATA IS NOT ABOUT THE DATA THE REVOLUTION IS NOT THAT THERE S MORE DATA AVAILABLE THE REVOLUTION IS THAT WE

More information

EXAM - AWS-Solution-Architect- Associate. AWS Certified Solutions Architect - Associate. Buy Full Product

EXAM - AWS-Solution-Architect- Associate. AWS Certified Solutions Architect - Associate. Buy Full Product Amazon EXAM - AWS-Solution-Architect- Associate AWS Certified Solutions Architect - Associate Buy Full Product http://www.examskey.com/aws-solution-architect- Associate.html Examskey Amazon AWS-Solution-Architect-Associate

More information

Welcome to the New Era of Cloud Computing

Welcome to the New Era of Cloud Computing Welcome to the New Era of Cloud Computing Aaron Kimball The web is replacing the desktop 1 SDKs & toolkits are there What about the backend? Image: Wikipedia user Calyponte 2 Two key concepts Processing

More information

Big Data Hadoop Stack

Big Data Hadoop Stack Big Data Hadoop Stack Lecture #1 Hadoop Beginnings What is Hadoop? Apache Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware

More information