Evolution of Inter-Cloud Computing to Handle Big Data. Tomonori Aoyama. Chair of GICTF Keio University

Transcription

1 5 th China Cloud Computing Conference 2013 Evolution of Inter-Cloud Computing to Handle Big Data June 7, 2013 Tomonori Aoyama Chair of GICTF Keio University

2 What is Inter-cloud? Inter-cloud computing allows on-demand reassignment of cloud resources including compute, storage and network, and transfer of workload through interworking of multinational and multi-operator cloud systems.

3 Evolution of Cloud Computing From Single Cloud to Hybrid Cloud, which is a connection between Public and Private Cloud, and then to global Inter-Cloud Computing Single Cloud(~2010) Hybrid Cloud (2011~) Inter Cloud (2015~ ) Government Organization Private Cloud Big Company Public Cloud SaaS/PaaS クラウド Provider General User Small Company Government Organization Private Cloud Big Company Public Cloud SaaS/PaaS クラウド Provider General User Small Company Private Cloud Public Cloud Network Public Cloud Private Cloud T. AOYAMA 3

4 Why? Three Data require Inter-Cloud 1. Big Data More than Zetta Byte Data are being produced. Enterprise Data, Video Data, Sensor Data, 2.Open Data Open Data Policy Personal Number Law (Japan) was approved. 3.Personal Data Medical Welfare Cloud for EHR & PHR My hospital anywhere is targeted. (Japan) I N T E R C L O U D

5 Annual global IP traffic will pass the zettabyte threshold by the end of 2016, Internet Video Will Drive Most Consumer Internet Traffic Through ftp P2P Video Web Sources: Cisco estimates based on CAIDA publications, Andrew Odlyzko 54% Internet video (86%, the sum of all forms of video (TV, VoD, Internet, and P2P) ) Sources: 5/ns827/VNI_Hyperconnectivity_WP.html T. Aoyama 5

6 Use case 1: Guaranteeing performance against an abrupt increase of the load Service consumer Avoid additional delay caused by the increased load by means of alternating data for access point to other cloud provider Service 2 Numerou s access attempts Internet Exchange data such as ID information of consumer Abrupt increase in the number of access attempts Lease the required quantity of resources from the community cloud overload Service 1 Provider X Network Distribute the load on the own servers to outside the own cloud system Service 1 Provider Y: Community cloud Return the processing to the original cloud system when load distribution becomes unnecessary

7 Use case 2 : Disaster Recovery Select a cloud system to be used for service recovery through negotiation with other cloud systems Remotely activate the application for Service 2 (some quality requirements are satisfied on a best-effort basis) Service 2 AP MW Cloud B Service X Service 2 Service 2 Disaster Service 1 Cloud A DBs are copied and synchronized in advance Service 1/2 Service user GICTF Service 1 DB DB AP MW Cloud D Remotely activate the Service 1 application for Service 1 (to guarantee the required quality) Service Y Cloud C Continue to provide services by recovering each service of Cloud A by using a cloud system other than that of Cloud A 7

8 GICTF: A forum for the Inter-cloud Technology Promote R&D on Inter-cloud technologies and the global standardization through collaboration between academia, government and industry Established on July 17 th 2009 Cloud system A Applications Standard inter-cloud interface Application Cloud system B Applications Servers Storage devices Network Internet, NGN, SDN. Member: 86 organizations ( NTT, KDDI, NEC, Hitachi, Fujitsu, Toshiba, Microsoft, IBM, Oracle, Cisco, VMware, IIJ, BIGLOBE, NICT, NII, NRI, etc. ), 38 members from Univ. 8

9 Key Issues for Inter-Cloud Service Federation Cloud System A (C) Provisioning, monitoring and control across multiple clouds Cloud System B Provisioning Provisioning Legacy System E-Gov Finance Medical CDN Application Server Storage Monitor and Control (A) Inter-cloud interworking architecture and standard interfaces Monitor and Control CDN e-gov Finance Medical (B) SLA assurance for totality of server, storage and NW Network Our target is an environment in which there are more than 10 distributed clouds, each consisting of several hundreds of applications and several thousands of virtual servers.

10 Functional Structure of Inter-cloud systems Provisioning Service provision requirements Provisioning Inter-cloud monitoring and negotiation control Resource plan Monitoring and control Search for available resources Policy negotiation Monitoring and control Dynamic reconfiguration of the server environment Resource reservation request Servers and storage Flexible resource allocation, reconfiguration Monitoring information Servers and storage Dynamic reconfiguration of the network environment Intranet Network configuration / optimization Network Intranet GICTF 10

11 The World s First Public International Standards Setting on Cloud Network 7 frameworks of public international standards are almost completed Proposal based on the results Major Deliverables I. Use case and functional requirements for Inter-Cloud Computing E/Aug 2010 II. Inter-Cloud interface specification on protocols J/Dec 2011, E/Apr 2012 III. Inter-Cloud interface specification on resources data model for network control J/Dec 2011, E/Apr 2012 IV. Network and technical requirements in support of Inter-Cloud J/Dec 2011 E/Apr I. Cloud Computing Definition Vocabulary definition, etc. II. Requirements Service requirements Use cases, etc. III. Cloud Network Network structures Network requirements, etc. IV. Cloud Computing Infrastructure Structures of infrastructure Infrastructure requirements, etc. V. IP Resources Management Ability of resources management, etc. VI. Inter-Cloud Method of inter-cloud Security management, etc. VII. Cloud Computing DaaS Requirements and architecture of Desktop as a Service 11

12 Inter-Cloud IF Framework of Inter-cloud Interface Specification The interface between two cloud systems administered by different operators Three layer modeling Lower layer protocols assumed as some XML message exchange, e.g., REST or SOAP Inter-cloud protocol: Information flows, message semantics with associated parameters specified Data models for network resources specified Data models for computing and storage referenced to other SDO s specifications Cloud System Inter-Cloud Protocol Cloud System Data Center Resource Data Model (Computing and Storage) Network Resource Data Model (Network) Inter-Cloud Protocol Data Center Resource Network Resource Data Center Resource Network Resource Lower Layer Protocols (e.g., REST or SOAP)

13 Cloud Standardization Status Taxonomy, Use cases, SLA, Requirements NIST ITU-T ISO/IEC JTC1 ETSI Open Cloud Manifesto KCSA CCF GICTF SIENA OGC CBA Users/ Operators Cloud Mng API OGF DMTF SNIA OpenStack Implementation SaaS API PaaS API IaaS API OASIS Architecture ITU-T NIST ATIS CSF Cloud BSS/ OSS ITU-T TM forum DMTF OCC Security NIST CSA ENISA ISO/IEC JTC1 SDO coordination OMG IEEE Other Cloud Inter-Cloud GICTF DMTF ITU-T IEEE OCC Global USA EU Asia Japan

14

15 Collaboration among GICTF and various SDOs DE-facto Standard OGF- Europe OGF SLG Special Liaison Group member OMG Cloud Interoperability Roadmaps Session US Government NIST CC forum DMTF SNIA De-jure Standard Open Source Community OpenStack ITU-T FG Cloud CC Standard Study Group Cloud Business Google, Salesforce, Amazon, etc. T. AOYAMA ISO/IEC JTC SC

16 Inter-Cloud System R&D* ( ) *Funded by Ministry of Internal affairs and Communications of Japan Application Server Storage Cloud System A AP AP AP Cloud System B Cloud Resource Provisioning Cloud Resource Federation and Reconfiguration Network Wide Area Network Dynamically Reconfigurable Network Physical World WAN edge Real-time Sensor Node

17 Inter-cloud test bed planned in 2012 Inter-cloud technology evaluation will start in 2012 on the national-scale test bed where three cloud data centers in Tokyo, Tohoku, and Kyushu will be connected with each other via JGN-X. Exclusive Line100M Cloud B Base(2) Tohoku JGN-X Tohoku-1 JGN-X Kyushu-1 Base(3) Kyushu Base(1) Tokyo JGN-X Tokyo-1 Cloud A JGN-X Kyushu-4 Cloud C

18 Necessity of Inter-Cloud Testbed Testbed for Cloud Networking will be an essential part of Inter-Cloud technology developments Coordinated developments through open test environments will be mutually profitable for both cloud computing and networking like New Generation Network International cooperation with various organizations in the industry for the efficient development of Inter-Cloud standards Promotion of cloud networking technologies and gaining the momentum Advanced Application Technology Inter-Cloud Technology R&D Testbed (domestic companies and universities) Cloud Computing Testbed New Generation Network Testbed JGN-X International cooperation IEEE Cloud Testbed companies & universities Cloud Reserch Institution Optical Networking Technologies Virtualized Network Wireless Networking Technologies Cloud Reserch Institution 18

19 Emerging Areas of Study in Future Network Research Introducing programmability into networking to flexibly and dynamically resolve constantly arising contemporary issues. Network Virtualization (NV) Software Defined Network (SDN) Network Functions Virtualization (NFV) And more? The University of Tokyo Confidential 19

20 SDN (Software Defined Network) and NFV ( Network Functions Virtualization) Publication at SDN and OpenFlow World Congress in October 22-24, 2012

21 SDN Standardization

22 DPN: Deeply Programmable Network Application Programmability ONF OpenFlow SDN Control-Plane Programmability Interfaces Functions Route Control Access Control Network Management Data-Plane Programmability Interfaces Functions Packet Data Processing Network Appliances (DPI, BRAS, EPC) In-Network processing (Cache, Transcode) Wide-Area generic processing Handling New Protocols IPvN (N>6), New Layer2, CCN OpenDaylight NFV DPN Deeply Programmable Network (DPN) Applications Control Plane Data Plane Network Applications Interfaces e.g, Northbound API Control-Plane Elements e.g, OpenFlow Controller Interfaces e.g, Southbound API Data-Plane Elements e.g, OpenFlow Switches The University of Tokyo Confidential 22

23 Architecture of VNode Prototype Users AGW Internet Domain Controller DC P User Opt-In R VNode P R Redirector Programmer P R Access Gateway (AGW) AGW Internet User Opt-In Users VNode Architecture Redirector (R) Form he Skelton of Virtualized Networks Redirector Manager Tunnel Redirection VNode Manager Slow Path Fast Path S P F P Programmer Manager S P F P S P F P S P F P S P F P Programmer (P) Programmability in Virtualized Networks Slice Traffic Production Router (10Gbps) Data Plane Forwarding Engines T. Aoyama 23

24 JGN-X Network Figure (from 2011/4) Realizing multiple New Generation Network Planes on a virtual JGN-X network. Also made available is to interconnect with other Testbeds. DCN Plane DCN DCN DCN OF OF Openflow Plane OF OF OF OF VN VN VN V-node Plane VN Ex 40G 10G 1G DF VN OF Virtual Node Openflow DCN DCN Virtual Storage StarBED VLAN Testbed Network Optical Testbed Wireless Testbed Sapporo JGN-X Planned POP Kashima DCN Hakusan Yokosuka Kobe Hokuriku Keihannna Tsukuba Kyusyu Univ. Kyusyu Inst. Tech Tohoku Univ. Tokyo Univ. Osaka Univ. Optical Testbed Koganei- Otemachi Otemachi~Otemachi To Korea 10G Fukuoka OF DCN Okayama Hiroshima OF 10Gx2 10G 10G Kanazawa 10G 10G 10Gx2 NICT Koganei 40Gx2 Osaka 40G 40G VN DCN OF OF Nagoya VN OF Wireless Testbed 1G Sendai OF 10G VN OF DCN Tokyo Hong Kong USA LA, USA Bandwidth International Japan- USA: 10G Japan - Korea: 10G Japan Hong Kong: 10G Hong Kong - Singapore 2.4G Singapore - Thailand: 622Mbps Singapore T. Aoyama Thailand 24

25 Social Cloud Infrastructure w/ SDN Security and Data Portability in Virtualization & SDN env.

26 Integration of Inter Cloud and Network Virtualization Versatile Information Services Total Handling and Control Mechanism Virtual Network Resource Virtual Server Resource Virtual Storage Resource T. Aoyama 26

27 Technical Issues on Network Virtualization for Intercloud Cooperation Resource Abstraction Hiding details of resources Name resolving of resources Resource Isolation Isolation among private virtual networks QoS control to guarantee bandwidth or latency Programmability Enabling creation of network functions based on new ideas (cache/encryption) Elasticity Providing network resources on-demand Cloud networking among clouds Acknowledgement & Credits: The white paper of Network virtualization study group in 2010.

28 Summary Inter-Cloud is a sequent trend toward future ICT systems GICTF objectives Promoting the open interface between cloud systems Contributing the Inter-Cloud standardization based on technical feasibility studies GICTF deliverables and contributions to SDOs Use case and functional requirements for Inter-Cloud Computing Technical Requirements for Supporting the Inter-cloud Networking Inter-Cloud Interface Specification Draft Cloud Resource Data Model Inter-Cloud Protocol Collaboration with DMTF and SNIA Support for joint projects and international intercloud testbeds

29 Global Inter-Cloud Technology Forum (GICTF)