The Choice of a Cloud Service Provider. Some (incomplete) Return-On-Experience. June 16th, 2014

Transcription

1 The Choice of a Cloud Service Provider Some (incomplete) Return-On-Experience INSPIRE Conference 2014 Workshop Sessions Benoit BAURENS, AKKA Technologies (F) June 16th, 2014 This content by the InGeoCloudS consortium members is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. Based on a work at INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

2 Public Clouds A quick evolving market Big Players Price war Amazon announced in March 30% decrease of their prices for CPU costs for its most-used instances types IBM acquired SoftLayer for $2 billion in 2013 and commits $1.2 billion to expand the SoftLayer cloud Footprint As for your holiday s renting, you re basically lookingfor: Good location Good price Good quality Trust INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

3 Typical Categories of Requirements to Consider We consider here the IaaS Delivery Model (Infrastructure as a Service) Location Business Continuity Technical Flexibility Commercial Offers Security Support Services INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

4 Typical Categories of Requirements to Consider Location Business Continuity Technical Flexibility Commercial Offers Security Support Services INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

5 «Location» Requirements Examples Guarantee data residency in some country and/or continent for legal constraints for performance reasons e.g. Data residency statements by IBM: Data storage location is the data center used by a customer to provision resources (e.g. a physical or hardware server). Whatever the storage solution chosen by the customer, data will be stored only in the data center in which resources are provisioned. The customer can also explicitly choose another data center when using storage services. IBM/SoftLayer does not replicate data over data centers, except for the Object Storage service (see below). In this case, data are replicated on data centers located in the same storage cluster (North America, Europe and Asia). INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

6 Typical Categories of Requirements to Consider Location Business Continuity Technical Flexibility Commercial Offers Security Support Services INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

7 «Business Continuity» Requirements Examples Guarantee data stored on the Virtual Machine s disks are not lost due to VM/hardware failure, for instance (i.e. disk RAID) CSP must provide solutions and best practices to achieve data persistence beyond the end of instances lifetime. Guarantee for high availability (e.g. hardware maintenance SLA) one among many others! INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

8 Typical Categories of Requirements to Consider Location Business Continuity Technical Flexibility Commercial Offers Security Support Services INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

9 «Technical Flexibility» Requirements Examples a varied catalog of instance types designed for performance in computing and/or disk I/O and/or network I/O. Type VCPU RAM DD NIC Tiny Mb 10 Gb 1 Small 1 1 Gb 10 Gb 1 Medium 1 2 Gb 10 Gb 1 Large 1 4 Gb 20 Gb 1 XLarge 1 8 Gb 20 Gb 1 XLarge.100Go 1 8 Gb 100 Gb 1 NB: Few CSPs actually offer real hardware on demand INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

10 «Technical Flexibility» Requirements Examples Family General Purpose Compute Optimized Insta Virtual CPU Memory (GiB) Instance EBS-Optimized Network nce CPU (ECU) Storage Performance Types M1 1 to 4 1 to to 15 1x160 to 4x420 not for all Low to high M3 1 to 8 3 to to 30 1x4 to 2x80 (SSD) not for all Moderate to high C1 2 to 8 5 to to 7 1x350 to 4x420 not for all Moderate to high CC x Gigabit C3 2 to 32 7 to to 60 2x16 to 2x320 (SSD) not for all Moderate to 10 Gigabit GPU Instances Memory Optimized Storage Optimized Micro Instances G x60 (SSD) yes High CG x Gigabit M2 2 to to to x420 to 2x840 not for all Moderate to high CR x120 (SSD) - 10 Gigabit I2 4 to to to to 8x800 (SSD) not for all Moderate to 10 Gigabit HI x1024 (SSD) - 10 Gigabit HS x2048 (SSD) - 10 Gigabit T1 1 Variable 0, Very low INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

11 «Technical Flexibility» Requirements Examples segmentation and isolation of private network(s), e.g. VLANs provisioning of ranges of consecutive public IP addresses provide a VPN service (preferred with IPSec) between your and CSP s infrastructures provide Firewall facility with customization on TCP/UDP and standard and custom port numbers provide a load balancing service that handles at least HTTP (TCP), HTTPS (TCP), INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

12 «Technical Flexibility» Requirements Examples Example of AWS for choosing best-fit storage methods some services (e.g. S3) are designed to reduce the risk of data lost whereas other services (e.g. Instance Store) focus on performances (high I/O) INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

13 Typical Categories of Requirements to Consider Location Business Continuity Technical Flexibility Commercial Offers Security Support Services INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

14 «Commercial offers» Requirements Examples provide relevant PaaS solutions (e.g. Web mapping software, postgres/postgis, Hadoop ) provide tools (e.g. Web console) and services (e.g. monitoring, API) to fully manage the infrastructure Single tenant environment: the ability to run instances on dedicated physical servers, with the guarantee that other customers do not run their instances on the same servers. This usually ensures better performance and improves security. Generally more costly Idem for network elements (virtual routers, firewalls ) INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

15 «Commercial offers» Requirements Examples single global point of contact to escalate technical and commercial issues provide PaaS-type services for data import and export. In particular, in order to backup data outside of CSP data center or in order to be able to eventually change of CSP in the future (Reversibility). Provide detailed billing: having a fine-grained, accurate and reliable view on the costs induced by usage of CSP services is a key requirement. Particularities of cloud accounting/billing versus more traditional hosting services necessitate careful and informed system design and deployment plan beforehand. INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

16 Typical Categories of Requirements to Consider Location Business Continuity Technical Flexibility Commercial Offers Security Support Services INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

17 «Security» Requirements Examples comply with certification SOC2 type2 (in North America) and ISO27001 in Europe trust APIs for user authentication and roles management: Robust, time-tested solutions and know-how brought by the CSP are often judged by consultants and security specialists as preferable as a self-made authentication service, prone to security leaks with regards to the target infrastructure. Propose Public-key cryptography (PKI): securely communicate on an insecure public network, reliably verify the identity of a user via digital signatures (source: Wikipedia) INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

18 Typical Categories of Requirements to Consider Location Business Continuity Technical Flexibility Commercial Offers Security Support Services INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

19 «Support Services» Requirements Examples provide an helpdesk reachable by phone or direct chat provide a web-based trouble ticketing system consultable by customers provide webtools for supervising/monitoring main indicators about the resources used such as system health, RAM/Disk usage, Peaks of load, I/O statistics This kind of supervision allows a fine-grained analysis of infrastructure usage and informed decisions for evolutions. Systematically get alerts or information messages about maintenance operations INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

20 Choice of the Cloud Computing Platform in the InGeoCloudS Project INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

21 InGeoCLOUDS Architecture INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

22 Choice of the Cloud Computing Platform in the InGeoCloudS Project Estimated resources: 13 instances, 2000GB storage, 20 GB/month network We analyzed several Cloud providers: Amazon AWS, SigmaCloud, Atlantic.Net, Flexiant Flexiscale, GoGrid, Google App Engine, Joyent, Microsoft Azure, OpSource, Rackspace, OVH Public Cloud. On the basis of several criteria: Functional/Software Requirements, Elasticity Model, As a Service Model, Maturity and Diffusion, Migration Cost Model Including Monthly Cost: E.g., Amazon AWS 500, MS Azure 600, RackSpace 1200 We observed average 10% costs drop in the 5 years INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

23 How to fairly bill users of a shared infrastructure? INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

24 Billing and Costs: Costs from Jan Jan 2013 Feb 2013 March 2013 April 2013 May 2013 June 2013 July 2013 August 2013 September 2013 October 2013 November 2013 December 2014 January 2014 February INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

25 Billing and Costs Goals: To split the cost across the different IGC services To split the cost across the different Data Providers We have a per Amazon service bill We would like to minimize it! It is not easy to map this bill to the different services, and to the different levels of usage If map from data provider A is accessed 10 times more than the map from data provider B, and B s map is 10 times larger, who should pay more? We implemented a fair billing, which is a useful tool for the definition of an actual business plan INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

26 Billing and Costs: Where do the costs come from? A typical Amazon bill looks like the following: Amazon Elastic Compute Cloud Amazon EC2 running Linux/UNIX $0.065 per M1 Standard Small (m1.small) Linux/UNIX instance-hour (or partial hour) 8,252 Hrs $0.260 per M1 Standard Large (m1.large) Linux/UNIX instance-hour (or partial hour) 966 Hrs $0.130 per M1 Standard Medium (m1.medium) Linux/UNIX instance-hour (or partial hour) 4,265 Hrs Amazon EC2 EBS $0.095 per GB-Month of snapshot data stored GB-Mo 6.49 $0.11 per 1 million I/O requests 31,370,334 IOs 3.45 $0.11 per GB-month of provisioned storage 3, GB-Mo Elastic Load Balancing $0.028 per LoadBalancer-hour (or partial hour) 690 Hrs $0.008 per GB Data Processed by the LoadBalancer GB 0.04 Amazon CloudWatch $0.50 per metric-month Metrics 2.12 It is quite complex: machine hours used but also I/O requests Fair sharing scenario: how to split costs among users (i.e. data providers)? INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

27 Billing and Costs: Questions How to relate a data provider action to a usage share of cloud resource? Easy: data provider executes some computation on an ondemand instance (e.g. shakemaps) Easy: data provider uses a portion of the total storage Less easy: web server shows a page of data provider Not so easy: CPU cycles/ios to serve a database query issued during the visualization of a given map of data provider General considerations: there are easy measurable resources, e.g., GBs of storage there are shared services and functionalities, e.g., API server, replicated Database server, etc. etc. Solution: Tag Amazon Resources Parse Amazon detailed billing report Include reserved instances cost Store resulting costs with the other indicators INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

28 Billing and Costs: Per service accounting INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

29 Billing and Costs: Per User accounting Detailed user s share of costs (service by service) Methodology: We defined 3 policies to share costs: A) proportional to usage. B) evenly. C) linear combination. For each entry in the Amazon bill, we defined which policy to apply Each entry in the Amazon bill is mapped to an IGC service Compute costs per data provider per service INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

30 Billing and Costs: Per User accounting In some cases, usage can be measured accurately By measuring directly storage occupancy (both DB and FS) In others it is not possible We introduced Global Usage Indicator : We assume that the number of map server requests is a good estimate of the IGC usage It measures if data from provider A is more used than data from B It approximates the ration of DB requests, FS requests, etc. The indicator is calculated by parsing map server logs It could be integrated with other sources e.g. DB logs, portal logs, Triple store logs, It is stored as a new indicator The cost of Shared resource is accounted according to the Global Usage Indicator INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

31 Billing and Costs: Per User accounting INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

32 Billing and Costs: Detailed Per User accounting INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

33 Billing and Costs: Observations about costs What we did wrong: We underestimated the development costs Even if some tests can be done out of the cloud (which is not free anyway), one should plan a twin deployment We took advantage of reserved instances late Motivation: it was not possible to modify the reserved instance type, this changed only in Oct. 2013, i.e. when we started using reserved instances. What we did well: Our early estimation was sufficiently correct On the basis of in premises configuration of data providers infrastructures Some resources were tuned over time Tuning in the type and number of instances From S:5, M:5, L:1 to S:10, M:2, L:1 INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,

34 Thanks for your attention Before leaving Have your copies of the Blueprints («How-To» series) and of latest newsletters. Take some goodies (Cups) And consider free trials on InGeoCloudS. INSPIRE Compliant Data and Services on the Cloud WORKSHOP, Monday June 16 th,