IaaS Migration Using the FELIX Federated Testbed

Transcription

1 2016 5th IEEE International Conference on Cloud Networking IaaS Migration Using the FELIX Federated Testbed Atsuko Takefusa, Jason H. Haga, Fumihiro Okazaki, Katsuhiko Ookubo, Seiya Yanagita, Ryousei Takano and Tomohiro Kudoh Information Technology Research Institute, National Institute of Advanced Industrial Science and Technology (AIST) Central 1, Umezono, Tsukuba-shi,Ibaraki , Japan Information Systems Architecture Sciences Research Division, National Institute of Informatics Hitotsubashi, Chiyoda-ku, Tokyo , Japan Information Technology Center, The University of Tokyo Yayoi, Bunkyo-ku, Tokyo , Japan Abstract The ability to migrate services to a remote data center is an important feature of Cloud services, especially after a natural disaster. The FELIX project has developed a global scale testbed by federating multiple, distributed, data centers, called SDN islands. FELIX leverages on a transit network resource manager (TN RM) to enable NSI and GRE connections between different SDN islands. In this study, we perform an IaaS migration over the EU-Japan FELIX federated testbed and NSI-enabled network. The experimental results show that IaaS migration in a low latency environment performs high-speed migration, while in a high latency environment, communication optimization achieves a reasonable 10-minute IaaS migration. Keywords-Software defined networking (SDN); Cloud computing; Network Services Interfaces (NSI); IaaS migration; federated testbed; resource management I. INTRODUCTION One important feature of Cloud services is the ability to migrate services to a remote data center especially during a natural disaster. After the Great East Japan Earthquake in 2011, many organizations created a business continuity plan for IT services. Cloud providers can generally provide high-availability services by using Cloud management technologies, such as redundancy and virtual machine (VM) migration, in a single data center. However, in the case of natural or artificial disasters that disrupt the electric power supply, administrator of two data centers are required to cooperate in order to maintain services. Infrastructure as a Service (IaaS) is a type of Cloud service that provides isolated resources in the form of isolated tenants consisting of virtual machines (VM), storage volumes, and networks. Multiple users can then create and deploy a variety of business related services on these tenants. IaaS management software can migrate virtual machines to another physical node in the data center because virtual machine disk images generally are kept in shared storage. However, in the case of a natural disaster, the entire IaaS including user virtual machines, shared storage, and management servers needs to be migrated to another remote data center. The FELIX (FEderated Test-beds for Large-scale Infrastructure experiments) project aims to provide researchers with an isolated experimental environment, called a slice, distributed over multiple data centers, called SDN (Software Defined Networking) islands [1], [2]. In order to achieve a Slice-based Federation Architecture (SFA) [3], FELIX developed a resource management software stack, and constructed a federated testbed spanning Europe and Japan SDN islands. As part of the software stack, we created a transit network resource manager (TN RM) to facilitate the provisioning of Network Services Interface (NSI) [4] enabled R&E networks and Generic Routing Encapsulation (GRE) tunnels over the Internet for transit network connections between SDN islands. Migration of services during a disaster is a multi-factor process that requires many different stages. This study, focused on an IaaS migration over FELIX by creating a slice over SDN islands in Japan and Poland, then migrating the IaaS environment using a NSI-enabled network. We also performed IaaS migration between two islands in AIST, Japan, using GRE tunnels as a transit network. The experimental results show that IaaS migration in a low latency environment performs high-speed migration as expected, while in a high latency environment; communication optimization is necessary to achieve a reasonable 10-minute IaaS migration. II. FELIX ARCHITECTURE The FELIX project is a collaborative effort between Europe and Japan to develop a management software stack that constructs a slice spanning international SDN islands, leveraging on NSI and GRE technologies for transit network connections between SDN islands. Fig. 1 shows an overview of FELIX architecture. The details of the architecture were described previously [1], however, some details of the architecture relevant to IaaS migration will be described in the following sections. A. FELIX Management Software Stack (MSS) The isolated experimental environment FELIX provisions for the user is a network slice and control of the SDN switch /16 $ IEEE DOI /CloudNet

2 Figure 1. FELIX Architecture. ((M)RO: (Master) Resource Orchestrator, C RM: Computing RM (Resource Manager), SDN RM: Software Defined Networking RM, SE RM: Stitching Entity RM, TN RM: Transit Network RM, AAA: Authentication, Authorisation & Accounting, (M)MS: (Master) Monitoring System) Figure 2. AIST island 1. allocated to that slice. As shown in Fig. 1 the SDN islands have resource orchestrators to coordinate users and resource managers (RM) using the GENI Aggregate Manager API, v3 (GENIv3) [5] for communication. Each SDN island consists of physical resources that are managed by different RMs. Details regarding the functionality of each RM were previously described [1]. A specific example of the SDN islands located at AIST is shown in Fig. 2. Relevant to the IaaS migration, it is important to note that the current C RM implementations use KVM and Xen as virtual machine managers. In order to achieve the migration, it was necessary to develop a KVM based C RM because Xen does not support nested virtualization and we confirmed in previous work, complete isolation by using KVM [6]. As shown in Fig. 3, the TN RM dynamically allocates and provisions a path between different SDN islands over a transit network using either NSI-enabled networks or GRE tunnels. This provides increased flexibility when additional islands would like to federate with FELIX infrastructure. Details on how these connections are made by TN RM are described in Section II-B. Figure 3. TNRM architecture. Table I DIFFERENCE BETWEEN GENI AGGREGATE MANAGER API AND NSI CONNECTION SERVICE. Command GENIv3 NSI CS Resource allocation allocate Reserve & ReserveCommit Extension of rsv. end time renew Reserve & ReserveCommit Provision provision - Link up poa geni start Provision Link down poa geni stop Release Termination delete Terminate B. Resource Management of Transit Network between SDN Islands 1) Path Provisioning based on NSI: NSI Connection Service (CS) defined by Open Grid Forum is a protocol to 34

3 request and provide network services between two Network Service Agents (NSAs), such as a requester agent and a provider agent, a requester agent and an aggregator, or an aggregator and a provider agent. NSI CS enables users to reserve, provision, release and terminate a path between two ports, called Service Termination Points, and query about the reserved connection. Users can specify reservation start and end time, bandwidth, latency, VLAN tag IDs, and required service parameters for the link. Cooperation with multiple NSAs in tree-based and chain-based manners obtains a path connected over multiple domains providing NSI-based network services. Because NSI CS and GENIv3 apply different state machines and communication protocols, it was necessary to ensure compatibility between the two. NSI CS is a SOAPbased two-phase commit protocol by asynchronous communication while GENIv3 is a one-phase XML-RPC protocol by synchronous communication and assumes on-demand service only. As mapped in Table I, when the TN RM receives GENIv3 allocate request, the TN RM sends both Reserve and ReserveCommit requests to an NSI aggregator in order to resolve the protocol difference. GENIv3 renew is a command to extend the allocation end time, which is mapped to Reserve and ReserveCommit requests, enabling the modification of reservation times of the reserved resource. The TN RM takes no action when GENIv3 provision request is sent. However, the TN RM confirms if the provision request is submitted when it receives poa geni start to maintain consistency with the other RMs. The remaining commands correspond one-to-one. In the FELIX testbed, the TN RM in AIST island 1 sends a path request to an NSI aggregator [7], and the aggregator sends the request to the related provider agents on the GLIF AutoGOLE testbed establishing an inter-sdn island path as shown in Fig. 3. The AutoGOLE testbed is maintained on a volunteer basis and consists of over 20 participating research and education network domains from around the world, such as Netherlight, GÉANT, icairstarlight), and JGN-X. 2) Path Provisioning based on GRE: Although an NSIbased path over the AutoGOLE testbed is of higher bandwidth and quality than the Internet, stable connectivity of each path is on a best effort basis due to the volunteerbased contributions of each network domain. Therefore, the TN RM was designed to also provide GRE tunnel-based path over the Internet as an alternative. GRE switches as shown in Fig. 3 were developed using an OpenFlow controller RYU [8] and Open vswitch [9], and the TN RM controls the traffic encapsulation via the RYU REST API. In our implementation, multiple paths between the same SDN islands that share one GRE tunnel are distinguished by different VLAN tags. One drawback to this approach is that the GRE-related data attached to each packet drastically reduces communication throughput due to packet fragmentation. To avoid this, optimization by using a smaller MTU size is Figure 4. Overview of steps in migrating IaaS. Source SDN island is on the left. Destination or remote island is on the right. required. III. IAAS MIGRATION A. Overview of IaaS Migration As a use case for disaster recovery, we performed an IaaS migration over the FELIX testbed based on the notion of Hardware as a Service (HaaS) [6]. Unlike general HaaS, which provides physical resources on demand, our proposed HaaS provides users with virtual resources as if they are physical resources. Therefore, the IaaS environment can easily and seamlessly extend onto the HaaS in a different data center when the resources managed by the IaaS are saturated. In this case, the user constructs an IaaS environment over the virtual environment using nested virtualization technologies. In previous work, we confirmed the feasibility of two-layered virtualization by using Linux Container and KVM although there is some degradation in the input/output performance [6]. In spite of this, the HaaS layer enables us to migrate the entire IaaS environment, including user virtual machines, shared storage, and management servers easily. Fig. 4 shows an overview of the IaaS migration, from source to remote SDN island, on FELIX using NSI-enabled transit network for slice extension. Here, a virtual machine provisioned by the C RM is called Layer 1 VM (L1VM), and a virtual machine provisioned by a FELIX user, namely IaaS manager, is called Layer 2 VM (L2VM). The steps of IaaS migration consist of: 1. Construction of IaaS in the source SDN island. Based on the IaaS manager s request (1), the HaaS coordinator provisions a slice including SDN links and virtual machines for IaaS (2). Then the IaaS manager deploys an IaaS environment on the L1VMs in the source SDN island (3). The virtual machines for IaaS, which consists of one management server virtual machine with storage and user virtual machines, are deployed as L2VMs. 35

4 2. Slice extension to the remote SDN island. When the IaaS manager requests IaaS migration because of a natural disaster (4), the HaaS coordinator prepares virtual machines and SDN links in the remote SDN island, transit network links between the islands, and then extends the IaaS L2 data plane (5). 3. Stop and transfer the IaaS environment. The IaaS manager instructs the management server to hibernate the user virtual machines and enter IaaS maintenance mode. Then, the disk image of the management server virtual machine is migrated to a L1VM in the remote island (6). 4. Completion of IaaS migration. The IaaS manager restarts the management server and instructs it to resume user virtual machines on the L1VMs in the remote SDN island. Importantly, traffic from outside of the FELIX testbed to the IaaS environment should also be migrated to the remote SDN island. Although we did not test this, the HaaS coordinator can also manage available global IP addresses in each SDN island and provide these addresses to Cloud service providers using the migrated IaaS environment. These service providers can then reconfigure DNS or notify clients of a new address to complete traffic rerouting. B. IaaS Migration Experiments over the FELIX Testbed As shown in Fig. 5, we performed two IaaS migration experiments on the FELIX testbed. In Experiment (1), we used AIST island 1 and 2 connected by a dedicated 1 Gbps network. In Experiment (2) AIST island 1 and the PSNC island were connected by NSI-enabled networks and the Internet. In Experiment (1) and (2), the latencies are 0.4 ms and 290 ms and the iperf communication throughputs are 940 Mbps (117.5 MB/s) and 220 Mbps (27.5 MB/s), respectively. CloudStack [10] was used to create a management server, primary storage and secondary storage for the IaaS environment. We provisioned two L1VMs on different physical nodes in the source data center, and deployed the management server and storage on a single L2VM provisioned on one of L1VMs. The management server registers the other L1VM as a host, which provisions user L2VMs. When the user virtual machines are stopped, their snapshots are stored in the secondary storage in the management server L2VM. These images are used to migrate the entire IaaS by copying the disk images to the remote SDN island and resuming them. 1) Demonstration of IaaS Migration: In Experiment (2), the international path allocation step in Section III successfully allocated and provisioned the NSI path shown in Fig. 6. A screenshot of the CloudStack control interface (Fig. 7) hosted by the L2VM resumed in the PSNC island, shows the L1VM provisioned in the PSNC island was successfully Figure 5. Experimental setup of IaaS Migration. registered as a host of user L2VMs. We also confirmed that user L2VMs created in AIST island 1 were resumed on the host, thus completing the IaaS migration. In a similar process, we also confirmed IaaS migration within a single site in Experiment (1). 2) Comparison of Transfer Throughputs for IaaS Migration: In the IaaS migration steps, transferring the disk image of the management server L2VM between the L1VMs in two different islands is a substantial task. The size of transferred disk image in our experiments was 13.7 GB. We compared the performance of both Experiment (1) and (2) using the NSI or GRE-based approaches with multiple optimization methods. We used general SCP and high performance SCP (HPN-SCP) [11]. In the case of HPN-SCP, TCP buffer size was set to eight times the general size. For reference, the physical computing nodes are Intel(R) Core(TM) i GHz (4 cores) and 8 GB memory in the AIST islands, and Intel(R) Xeon(R) CPU 2.40GHz (24 cores) and 48 GB memory in the PSNC island. L1VMs provisioned in each island were allocated 4 cores, 4 GB memory, and had a MTU size of 1450 bytes. Table II shows the communication throughputs in Experiment (1) and (2). In the case of NSI-enabled network of Experiment (1), a lowlatency environment, the compression process becomes the bottleneck of the overall throughput because the CPU load of the sender node is saturated. HPN-SSH without encryption and compression achieves 83 % of the iperf3 throughput. Using a packet pacing method by PSPacer [12] shows comparable performance with stable throughput. When GRE tunnels were used in Experiment (1), the performance was generally worse than the NSI case and was especially poor when using HPN-SCP without encryption and compression. This severe performance degradation is reproducible in our environment and caused by lack of compatibility between HPN-SCP optimization methods and GRE tunneling pro- 36

5 by multiple sites through GTS management software. GTS isolates network slices by SDN switch ports, while FELIX isolates the resources by a combination of ports and VLANs. Both SEP and GÉANT have not performed experiments of real world use cases such as IaaS migration as of yet. Figure 6. Snapshot of NSI Viewer. Path is between AIST and JGN-X in Japan, icair in the United States, and Netherlight and PIONIER (GÉANT) in Europe. Figure 7. Snapshot of CloudStack GUI hosted in the PSNC island. cessing. In the case of NSI-enabled network of Experiment (2), a high-latency environment, HPN-SCP with compression yielded 87 % of iperf3 throughput and 6.46 times faster throughput than normal SCP. In this case, the elapsed time of data transfer between AIST, Japan and PSNC, Poland is 9 minutes and 34 seconds, which is reasonable for a transcontinental IaaS migration between Japan and Europe. When GRE tunnels were used in Experiment (2), the HPN- SCP result with encryption improves SCP throughputs, but the results using HPN-SCP without encryption show poor performance again. IV. RELATED WORK For multi-domain slice federation, Slice Exchange Point (SEP)-based architecture was proposed and developed in [13], which can convert between slice operations defined by different domains. SEP supports interoperation between VNode [14] and ProtoGENI [15] and enables slice creation over different domains, but it does not support dynamic provisioning of a transit network as in the FELIX infrastructure. GÉANT also proposed a federation testbed called GÉANT Testbed Service (GTS) [16] and defined an API that enables control of computing resources and SDN switches provided V. CONCLUSION We developed a TN RM to manage inter-connection between distributed SDN islands, and dynamically provision NSI-based paths or GRE tunnels. We also successfully implemented and demonstrated the migration of an entire IaaS environment across intercontinental SDN islands through the FELIX infrastructure. Moreover, we presented that the migration could be completed within a realistic migration time. Our previous work investigated nested virtualization for HaaS and discovered that although feasible, the overhead associated with KVM nested virtualization technology is not negligible [6] and is not suitable for practical IaaS operations. In spite of this limitation, this study provides important performance measurements when performing an IaaS migration using nested virtualization technology. We expect in the future that GENIv3 will support operations for virtual machine migration and that both nested virtualization technologies and migration technologies will improve, especially using bare-metal Cloud. It is important to note that the disk image size of a largescale IaaS environment is variable and will directly impact the migration time. The size of our IaaS environment is small, but the time to migrate this environment provides a reasonable estimate of how long larger environments will take. As a first approximation, assuming the migration time is linearly proportional to the size, if a single machine environment is on the order of TB (i.e. 100 times larger) the time to migrate would be 1000 min ( 17 hours). From our own experience, datacenters can have backup electrical generators that operate for hours, thus a 17 hour migration is still a feasible timescale following a disaster. Although optimization methods can reduce performance degradation due to high latency, they cannot completely overcome the performance of low bandwidth links. This reinforces the fundamental need to build high quality, high bandwidth transit networks in federated testbeds for carrying out experiments similar to the one described here, without disrupting production level systems. Future work will use these types of experimental network infrastructures to explore the re-routing of existing network connections. This study again demonstrates the need for high bandwidth and performance-assured network infrastructures in the future. ACKNOWLEDGMENT We would like to thank the FELIX and GLIF AutoGOLE members for supporting our international experiments. This work was partly funded by the EU-Japan FELIX project of the National Institute of Information and Commu- 37

6 Table II COMPARISON OF INTER-SDN ISLAND TRANSFER THROUGHPUT OF THE MANAGEMENT SERVER L2VM IMAGE. THE BASELINE THROUGHPUTS ARE MB/S AND 27.5 MB/S INEXPERIMENT (1) AND (2), RESPECTIVELY. VALUES WITH * INDICATE RESULTS USING SMALLER TRANSFER DATA AS REFERENCE. Experiment Communication Encryption Compression Packet NSI GRE pacing Rate [MB/s] Transfer time Rate [MB/s] Transfer time SCP m03s m04s (1)AIST1 SCP m30s m31s AIST2 HPN-SCP m56s m26s HPN-SCP m13s 0.086* HPN-SCP m33s m40s HPN-SCP m11s SCP m30s m58s SCP m17s m05s (2)AIST1 HPN-SCP m55s m07s PSNC HPN-SCP m55s 1.9* HPN-SCP m34s 2.7* HPN-SCP m16s nications Technology (NICT), Japan, and the Commission of the European Union, and the Project for Developing Innovation Systems of MEXT, Japan. REFERENCES [1] C. Fernandez, C. Bermudo, G. Carrozzo, R. Monno, B. Belter, K. Pentikousis, U. Toseef, T. Kudoh, A. Takefusa, J. Haga, B. Puype, and J. Tanaka, A recursive orchestration and control framework for large-scale, federated SDN experiments: the FELIX architecture and use cases, International Journal of Parallel, Emergent and Distributed Systems, pp. 1 19, [2] U. Toseef, C. Fernandez, C. Bermudo, G. Carrozzo, R. Monno, B. Belter, K. Dombek, L. Ogrodowczyk, T. Kudoh, A. Takefusa, J. H. HAGA, T. Ikeda, J. Tanaka, and K. Pentikousis, Implementation of the FELIX SDN Experimental Facility, in Proc. EWSDN 2015, , pp [3] L. Peterson, R. Ricci, A. Falk, and J. Chase, Slice-based Federation Architecture (SFA) v2.0, pp. 1 18, [4] G. Roberts, T. Kudoh, I. Monga, J. Sobieski, J. MacAuley, and C. Guok, NSI Connection Sevice v2.0, in GFD. 212, , pp [5] GENI Aggregate Manager API, wiki/gapi AM API. [10] Apache CloudStack, [11] High Performance SSH/SCP, php/hpn-ssh. [12] R. Takano, T. Kudoh, Y. Kodama, and F. Okazaki, Highresolution Timer-based Packet Pacing Mechanism on the Linux Operating System, IEICE Transactions on Communications, pp , [13] T. Tarui, Y. Kanada, M. Hayashi, and A. Nakao, Federating Heterogeneous Network Virtualization Platforms by Slice Exchange Point, in Proc IFIP/IEEE International Symposium on Integrated Network Management (IM 2015) (Poster), 2015, pp [14] A. Nakao et al., Advanced Network Virtualization: Definition, Benefits, Applications, and Technical Challenges, Version 1.0, nv-study-group-white-paper.v1.0.pdf, Network Virtualization Study Group, Tech. Rep., [15] R. Ricci and J. Duerig and L. Stoller and G. Wong and S. Chikkulapelly and W. Seok, Designing a Federated Testbed as a Distributed System, in Proc. TridentCom 2012, 2012, pp [16] GTS (GÉANT Testbed Service), GTS/Pages/Home.aspx. [6] R. Takano, A. Takefusa, H. Nakada, S. Yanagita, and T. Kudoh, Iris: An Inter-cloud Resource Integration System for Elastic Cloud Data Centers, in Proc. CLOSER 2014, 2014, pp [7] A. Takefusa, T. Kudoh, R. Takano, H. Nakada, K. Ookubo, and F. Okazaki, Development of a Reference Implementation of the NSI Connection Service Protocol, version 2.0, National Institute of Advanced Industrial Science and Technology (AIST), AIST14-A00014, [8] RYU SDN Framework, [9] Open vswitch, 38