Performance Evaluation of Live Migration based on Xen ARM PVH for Energy-efficient ARM Server

Transcription

1 Performance Evaluation of Live Migration based on Xen ARM PVH for Energy-efficient ARM Server Jaeyong Yoo, Sangdok Mo, Sung-Min Lee, ChanJu Park, Ivan Bludov, Nikolay Martyanov Software R&D Center Samsung Electronics

2 Contents Motivation Live Migration in Xen ARM PVH Design and Implementation Performance Evaluation 1. Streaming service with ARM vs. x86 2. Streaming server consolidation with live migration 3. Streaming service with quad-core ARM board Concluding Remark

3 Motivation

4 Energy Problem in Datacenters Datacenters eat up magnificent amount of electricity Space (17%) Racks (3%) Electricity (27%) Cooling Equipment (6%) Service (13%) Power Equipment (17%) Datacenter operation cost Engineering & Installation (19%) Ref: Jaroslav Rajić, ``Evolving Toward the Green Data Center,

5 ARM Servers for Future Green Data Center Economical choice Significant advantage in compute/watt Vendors of ARM Server Soc AMD: Seattle (64-bit ARM server processor, 2H 2014) Calxeda: ECX-1000 Applied Micro: X-Gene OS for ARM Servers Linaro LEG Redhat deploys ARM-Based Servers for Fedora Project AMD Seattle: 64-bit ARM server Applied micro X-Gene Calxeda Energy Core ECX-1000

6 ARM Servers for Future Green Data Center Economical choice Significant advantage in compute/watt Vendors of ARM Server Soc AMD: Seattle (64-bit ARM server processor, 2H 2014) Calxeda: ECX-1000 Applied Micro: X-Gene OS for ARM Servers Linaro LEG Redhat deploys ARM-Based Servers for Fedora Project AMD Seattle: 64-bit ARM server Further energy efficiency maximization: Server consolidation by virtualization Calxeda Energy Core ECX-1000 Applied micro X-Gene

7 Design and Implementation of Live Migration in Xen ARM PVH

8 Overall Architecture Components for Live Migration in Xen ARM PVH Dom0 libvirt perform-migrate Kernel xl libxl libxc ARM-migrate DomU apache mysql Kernel streaming server suspend /resume Hypervisor HVM context save/restore VCPU save/restore Memory map get/set Memory data save/restore dirty-page detecting get dirtybitmap Legend Existing module Newly Impleme nted Hardware (Arndale) Cortex-A15 Dualcore 1.7 GHz, 2GB Memory, SATA3, USB3.0 Modified module

9 Sequence of Live Migration migration source migration destination DomU Suspend VCPU save HVM save dirty bitmap dirty detection memory save memory get map xc domain - save xl migratereceive xl xc memory set map domain -restore memory restore HVM restore VCPU restore DomU resume get/set memory map store dirtypages start dirtypaging get dirty bitmap save/restore memory contents loop until stop-condition suspend domu last-dirty pages save/restore HVM save/restore VCPU resume DomU

10 Major Hypercalls for Live Migration Implemented Hypercalls for Enabling Live Migration Feature in Xen ARM PVH Functions Hypercalls Description Memory Migration XENMEM_get/set_memory_map Save/restore physical memory map of DomU XEN_DOMCTL_shadow_op XENMEM_add_to_physmap_range Enable dirty-page detection Get dirty-page bitmap Access the domu s memory from dom0 VCPU Migration XEN_DOMCTL_get/setvcpucontext Save/restore the vcpu registers HVM Migration XEN_DOMCTL_get/sethvmcontext Save/restore the hvm contexts (e.g., timer, interrupt controller)

11 Dirty-page Tracing: Get-dirty Bitmap libxc ARM-migrate XEN_DOMCTL_ shadow_op (peek dirtypages) hypercall param from toolstack: dirty-page bitmap get dirty-page bitmap Filling up the dirty-page bitmap Temporary dirty-page storing dirty pages Dirty-page detecting candidates: 1. Embedded in page table (use un-used bits in PTE) 2. Linked list of PFNs 3. Bitmap of PFNs

12 Dirty-page Tracing: Dirty-page Detection guest VA Guest page table Level 1 Level 2 Level 3 IPA domu kernel Level 1 Level 2 Xen page table Level 3 Level 1 p2m: physical to machine page table Level 2 Level 3 Xen-side for Xen itself MA Xen-side for domu

13 Dirty-page Tracing: Dirty-page Detection guest VA Guest page table Level 1 Level 2 Level 3 IPA domu kernel w=0 Level 1 Level 2 Xen page table Level 3 Level 1 Level 2 Level 3 PTE write bit=0/1 Xen-side for Xen itself MA Xen-side for domu

14 Dirty-page Tracing: Dirty-page Detection guest VA write request Guest page table Level 1 Level 2 Level 3 IPA domu kernel w=0 Level 1 Level 2 Xen page table Level 3 Xen-side for Xen itself MA Level 1 Level 2 fault traped by xen Level 3 PTE write bit=0/1 Xen-side for domu

15 Implementation Choice Manual walking of p2m table Virtual-linear page table

16 Manual Walking of p2m Table IPA Xen-side for Xen itself Level 1 Level 2 Level 3 Level 1 Xen-side for domu Superpage checking Level 2 PTE w bit modification PTE Level 3 MA create a mapping to Xen (3 times) physical memory (a.k.a. machine memory)

17 Virtual-linear Page Table Consider third-level page table as a continuous memory block in virtual address space virtually continous third-level page table (8GB DomU requires 16MB third-level page table) virtual memory Xen page table Lev el 1 Lev el 2 Lev el 3 3lvl PT #2 guest s third-level page table 3lvl PT #1 3lvl PT #5 physical memory (a.k.a. machine memory) ref:

18 Virtual-linear Page Table Consider third-level page table as a continuous memory block in virtual address space virtually continous third-level page table (8GB DomU requires 16MB third-level page table) virtual memory for given IPA, with some arithmetic, calculate the Xen VA and just read it! Xen page table Lev el 1 Lev el 2 Lev el 3 3lvl PT #2 guest s third-level page table 3lvl PT #1 3lvl PT #5 physical memory (a.k.a. machine memory) ref:

19 Evaluation

20 Experiment Environment (Hardware/Software) x86 hardware 8 cores (i GHz) Intel 1Gbps NIC 4GB memory ARM Arndale board 2 cores 1Gbps Network card (USB 3.0) SSD msata 2GB memory Xen source: Xen 4.4 staging Domain kernels: Dom0: Linaro kernel 3.11 DomU: Linaro kernel 3.9 Streaming server: ffserver (RTSP streaming) Exp. Platform 1 Exp. Platform 2 Streaming Server Linux Streaming Server Linux Exp. Platform 2 Streaming Server Linux xen x86 HW Arndale board clients 1G switch 220v power power source Power meter (Yokogawa WT3000)

21 Experiment Environment (Hardware/Software) x86 hardware 8 cores (i GHz) Intel 1Gbps NIC 4GB memory ARM Arndale board 2 cores 1Gbps Network card (USB 3.0) SSD msata 2GB memory Xen source: Xen 4.4 staging Domain kernels: Dom0: Linaro kernel 3.11 DomU: Linaro kernel 3.9 Streaming server: ffserver (RTSP streaming) Exp. Platform 1 Exp. Platform 2 Streaming Server Streaming Server Linux Linux Note: Major evaluations are performed within mobile-featured ARM board. Performance evaluation of server-featured ARM board is presented at the x86 end of the slides. HW Arndale board Exp. Platform 2 Streaming Server Linux xen clients 1G switch 220v power power source Power meter (Yokogawa WT3000)

22 Experiment Environment (Scenarios) Test case 1: Streaming service with ARM vs. x86 Saturate the streaming server to get the maximum number of streaming clients Test case 2: Streaming server consolidation with live migration 10% of the maximum number of streaming clients Measurement 1: Maximum number of streaming clients for each test platform Measurement 1: Energy-efficiency comparison for each test platform Measurement 3: Total live migration time, service downtime Measurement 2: Energy-efficiency comparison for each test platform Measurement 2: Streaming server consolidation within xen-virtualized servers Measurement 4: Dirty-page detection time, dirty-page get-bitmap time, total dirty-page counts Test case 3: Streaming with quad-core ARM board Maximum clients with varying number of ARM cores (in-progress)

23 Case 1: Streaming Service ARM vs. x86 (Maximum capacity of ARM virtualized Server) Max streaming clients with varying number of VMs Dual-core ARM board Single VCPU for each VM Number of VMs Per VM Memory Max Streaming Clients Watt 1 512MB around ~ MB around ~ MB around ~ MB around ~ Finding: ARM cores are major bottleneck point

24 Case 1: Streaming Service ARM vs. x86 (Energy-efficiency comparison to x86 hardware) Compare with the best case of ARM* virtualization OS Total memory in server Max Streaming Clients Watt Client/Watt Required memory x86 with Linux ARM with native Linux ARM with virtualization 4GB ~ W 6.17 CPW ~ 2.4GB 2GB ~ W CPW ~ 707MB 512MB ~ W 7.43 CPW ~ 340MB * Dual-core ARM CPU Finding: Even dual-core ARM with virtualization show higher CPW than x86

25 Case 2: Streaming Server Consolidation of ARM virtualized server Scenario: 4 ARM boards, each running a 256MB VM Each VM has 10 clients Consolidate all VMs to one ARM board, and turn off other 3 ARM boards Watts before consolidation Watts after consolidation Energy saving percentage 2 to 1 consolidation [extrapolated] 3 to 1 [extrapolated] 4 to 1 2 x 8w = 16w 8.6w 46% saving 3 x 8w = 24w 8.9w 63% saving 4 x 8w = 32w 9.4w 71% saving Finding: Server consolidation can significantly save energy consumption

26 Case 2: Live Migration Performance Migrate a VM at a time With different domu memory size (128MB, 256MB, 512MB) Measurements: Live migration time Whole time for live migration Total dirty pages Number of dirtied pages during the time of live migration

27 Case 2: Live Migration Performance Number of dirty-pages in iterations configuration for stop-condition max iter: 29 max_mem_factor: 3 min_dirty_per_iter: 50

28 Case 2: Service downtime due to live migration Service downtime The time that VM is not responding to outside interaction Measurement method: flood-ping to migrating domain time difference between packets send from the migrating domain

29 Case 2: Performance of dirty-page detection Measure the elapsed time of two major functions dirty-page detection dirty-page collection

30 Case 3: Quad-core ARM board (In-progress) ARM board: 4 ARM cores with 8GB memory Number of VMs Per VM Memory Max Streaming Clients Watt CPW 1 1GB ~ W 7.06 CPW 2 1GB ~ W CPW 3 1GB ~ W CPW x86 case: (see slide 24) OS Total memory Max Streaming Clients Watt Client/Watt x86 with Linux 4GB ~ W 6.17 CPW

31 Concluding Remark ARM server is a good candidate for green data centers Even ARM mobile processors with virtualization results in better CPW compared to x86 Virtualization in ARM servers can leverage the energy efficiency by server consolidation Pass-through to DomU could significantly increase the performance