Emerging Testbeds for NwGN through Virtualization Technologies. Akihiro NAKAO University of Tokyo NICT

Transcription

1 Emerging Testbeds for NwGN through Virtualization Technologies Akihiro NAKAO University of Tokyo NICT

2 Testbeds through Net Virtualization Various ideas under development GENI 5 Clusters PlanetLab Today s Talk: Emulab Mostly for Network Services OneLab2 Not for Network Architectures Federika CoreLab (ACM ROADS 2008) Mobitopolo (ACM SIGCOMM VISA 2009) PeerLab Virtual Router on JGN Aki NAKAO, NwGN meeting at Hiroshima 2

3 CoreLab

4 PlanetLab The largest and most popular overlay network test bed Currently consists of 900+ nodes at 450+ sites 800+ Projects/ Researchers 180+ high level academic publications enabled Aki NAKAO, NwGN meeting at Hiroshima 4

5 Brief History of PlanetLab PlanetLab 1.0 (2002 3) UNIX account slivers PlanetLab 2.0 (2003 4) Vserver slivers PlanetLab 3.0 ( ) PLCAPI 1.0 PlanetLab 4.0 (2007 ) MyPLC1.0 Federation Idea PlanetLab 4.2 (2008 9) Federation (PLE, PLJ, (PLK, PLC,=>PLA?)) New Development using Private PlanetLab We are here (RSpec/GENI wrapper, New Node Type, Monitoring, QA,VINI) Aki NAKAO, NwGN meeting at Hiroshima 5

6 What have we learned from PlanetLab? Concept Slice based experimental facility Slice Mechanism Resource Isolation through Virtualization Management Framework Node : Physical Machine Sliver : Virtual Machine Slice : A collection of Virtual Machines RSpec: Resource Description PLC: Central Management Servers Aki NAKAO, NwGN meeting at Hiroshima 6

7 Slivers Sliver Sliver Sliver Sliver Node Mgr Owner VM VM 1 VM 2 VM n Auditing service Monitoring services Brokerage services Provisioning services Virtual Machine Monitor (VMM) Linux kernel (Fedora Core) + Vservers (namespace isolation) + Schedulers (performance isolation) + VNET (network virtualization) - From Aki NAKAO, NwGN meeting at Hiroshima 7

8 PlanetLab Architecture Slice1 Slice2 Sliver Sliver PLC Node Sliver Sliver Node Aki NAKAO, NwGN meeting at Hiroshima 8

9 Limitations in PlanetLab Flexibility is sacrificed over Scalability/Performance Sliver : Container based Resource Isolation Shares crucial resources Single kernel (Linux with VServer patch) Network stack and modules (NAT, Firewall, etc) Network resources (IP address and ports) Vnet (virtual network device ; need work for multi homing) Cannot run arbitrary kernel/network stack & modules Software Engineering Issue Cannot keep up with the latest kernel (VServer) The latest VServer patch (PlanetLab ) Misses out support for new devices/chip features Aki NAKAO, NwGN meeting at Hiroshima 9

10 Design Principle for a New Testbed Performance Network I/O Scalability The number of slices Isolation Resource must be isolated per slice Flexibility Arbitrary kernel/network stacks/resources Code Reusability Catch up with the latest technology Minimize software engineering effort Largely Disregarded Aki NAKAO, NwGN meeting at Hiroshima 10

11 VEE Candidate for Sliver Implementation Apps Apps Manager MngOS Apps GuestOS apps GuestOS Apps GuestOS Apps GuestOS Apps VE VE VM VM VM Apps VM VM HostOS VMM HostOS Hardware Hardware Hardware Resource Container Hypervisor-Based Host-Based Design Principles Performance Scalability Security Isolation Performance Isolation Flexibility Code Reusability Resource Container advantageous disadvantageous Hypervisor Hosted VMM?? +? + + room for improvement Aki NAKAO, NwGN meeting at Hiroshima 11

12 CoreLab Design Sliver VEE : Hosted Virtual Machine Monitor VM Disk Image Transfer : Multipath & Caching Network Configuration : Multihoming Management Tools: Reuse PlanetLab s design (GENI wrapper/federation possible) Comparison Chart PlanetLab CoreLab Sliver Resource Container Hosted VMM Kernel Shared Separated Guest OS Various Linux Distros Any OSes IP & Port Shared Shared (managed) Privileged System Calls Shared Isolated Aki NAKAO, NwGN meeting at Hiroshima 12

13 CoreLab Prototype Implementation Sliver VEE : KVM + QEMU on Kernel VM Disk Image Transfer : BitTorrent Network Configuration : TunTap / Managed IP/Port Management Tools: MyPLC Optimizations Compression of VM Image Increase Sharable Buffers Decrease Influence of VM Boots Misc. SSH Login (permitopen, authorized_key) and VNC Compatibility with PlanetLab (PLCAPI + CoreLab APIs) Aki NAKAO, NwGN meeting at Hiroshima 13

14 KVM Hosted Virtualization Full/Para Virtualization (Virtio) Aki NAKAO, NwGN meeting at Hiroshima 14

15 Network Configuration PlanetLab Model CoreLab Model Aki NAKAO, NwGN meeting at Hiroshima 15

16 VM Image Dissemination Introduce helper nodes BT Node Master BT Node Slave Introduce components BTManager BTSeed Aki NAKAO, NwGN meeting at Hiroshima 16

17 Scalability Check (1/2) pkt size=64b Total packet receive rate (Kpps) Probably our system is CPU bound QEMUs nicely schedule by Linux Kernel Max at 48kpps (slightly worse than Xen by Trellis) Little overhead in term of # of VMs ( VMs) Great resource fairness among VMs (~1.8% deviation) Offered Load via pktgen (Kpps) Aki NAKAO, NwGN meeting at Hiroshima 17

18 CoreLab Deployment (on going) Phase1 : 12 Sites over JGN2Plus + SINET + others 10 in operation Phase2 : Asian Region HP DL 580 (4U) Intel Xeon 16-core (quad 4-core) 128GB (Max 256GB) memory 300GB RAID5 (accommodate up to 254 slices) Aki NAKAO, NwGN meeting at Hiroshima 18

19 CoreLab : In Action Aki NAKAO, NwGN meeting at Hiroshima 19

20 Conclusions Contributions Identify design principles for alternative test beds Performance, Scalability, Isolation Flexibility, Code Reusability Implement prototype test bed CoreLab Hosted VMM as sliver to achieve Report various optimization and preliminary evaluations Throughput 30 70% of Vanilla Linux and PlanetLab OS Future Work Build Full Fledged CoreLab Optimize performance and scalability Virtual Router via Hosted VMM Aki NAKAO, NwGN meeting at Hiroshima 20

21 More Future Directions CoreLab Enhancement Geographically (Japan to Asia) Public PlanetLab J Lab as a regional Public PlanetLab Federation Federate CoreLab, J Lab with PlanetLab, OneLab2 GENI Spiral 1 Solicitation 2 Virtual Router Slow Path Component Extreme Enhancement to CoreLab OpenFlow / NetFPGA Aki NAKAO, NwGN meeting at Hiroshima 21

22 Mobitopolo

23 Virtual Infrastructure for Testbeds Increasing availability of hosting environment: PlanetLab, CoreLab, EmuLab, Amazon EC2 New challenges: Consistent execution environment across heterogeneous hosts Live migration between hosts Maintaining connections between components during migration Aki NAKAO, NwGN meeting at Hiroshima 23

24 Consistent Execution Environments User Mode Linux (UML) Runs inside of virtual environments provided by PlanetLab (VServer), CoreLab (KVM), Amazon EC2 (XEN) Full Linux kernel functionality Execution environment for standard Linux binaries Added Ethernet/UDP tunnels Aki NAKAO, NwGN meeting at Hiroshima 24

25 Live Migration Scrapbook for User Mode Linux (SBUML) Extension of UML Provides VM Snapshots Added Live Migration Iterative copy while VM is still running Final copy with VM frozen Aki NAKAO, NwGN meeting at Hiroshima 25

26 Maintaining Connections Central Control Software Automatic Deployment VMs initialized from snapshots Tunnels automatically configured Automatic reconnection of UDP connections after migration Tunnels configured outside guest VMs, making it invisible to software inside Aki NAKAO, NwGN meeting at Hiroshima 26

27 Result: Mobitopolo User Mode Linux + SBUML + Ethernet/UDP Tunnels + Central Control Distributed Applications see Linux OS connected by Ethernet Aki NAKAO, NwGN meeting at Hiroshima 27

28 Experiment with Mobitopolo (in Florida) (in Tokyo) If protocol VM is in Tokyo, file copy BW = 130Kbps If protocol VM is in Florida, file copy BW = 790Kbps Aki NAKAO, NwGN meeting at Hiroshima 28

29 Implementation: UML/SBUML Linux ported to Linux system calls Block Devices simulated with large host files RAM simulated with memory mapped files MMU simulated with host processes SBUML puts host processes in stable state, saves RAM and Block Device files with tar. SBUML adds hooks for demand fetching disk blocks using HTTP Aki NAKAO, NwGN meeting at Hiroshima 29

30 Implementation: Tunnels UML provides TUN/TAP interface Point to point Ethernet from guest eth(n) device to host Tap device (requires root to create tap device) Host forwarding and routing connect tap device to outside network (requires root privileges) Mobitopolo modifies UML device driver to connect to UDP socket, instead of Tap. Benefits: no root privileges needed no interference by host routing supports any protocol on top of Ethernet Aki NAKAO, NwGN meeting at Hiroshima 30

31 Implementation: WAN Live Migration Iterative copy of running VM, each pass copying smaller and smaller delta, VM frozen on final pass Typical implementations (e.g. VMware, Xen, KVM) are LAN only solutions, so only copy RAM. WAN migration requires copying RAM and DISK. SBUML modifies tar to do block level iterative copying, which grabs both RAM and DISK Aki NAKAO, NwGN meeting at Hiroshima 31

32 WAN Migration (first draft implementation) 7.5 minutes downtime from Tokyo to Virginia over 6.2Mbps link Difficult migration due to high VM load in Tokyo undermining pre copy effectiveness 28 seconds downtime from Virginia to Florida over 8.0Mbps link (Note: Most WAN migration is tested on 100Mbps or 1Gbps links) Aki NAKAO, NwGN meeting at Hiroshima 32

33 More Data Aki NAKAO, NwGN meeting at Hiroshima 33

34 Related Work (User Mode Networking) Bavier, Feamester, Huang, Peterson, & Rexford: In VINI Veritas: Realistic and Controlled Network Experimentation Jiang, & Xu: Violin: Virtual Internetworking on Overlay Infrastructure. Both used UML 2 nd used custom UDP tunnels Neither had snapshots or migration Aki NAKAO, NwGN meeting at Hiroshima 34

35 Example Distributed Application Aki NAKAO, NwGN meeting at Hiroshima 35

36 Conclusion Standard Linux functionality and networking interface Portable user mode implementation Live migration across WAN Deployment of preconfigured VM snapshots and network topologies Physical deployment details transparent to distributed system Aki NAKAO, NwGN meeting at Hiroshima 36

37 Summary Various ideas under development GENI 5 Clusters PlanetLab Today s Talk: Emulab Mostly for Network Services OneLab2 Not for Network Architectures Federika CoreLab (ACM ROADS 2008) Mobitopolo (ACM SIGCOMM VISA 2009) PeerLab Virtual Router on JGN Aki NAKAO, NwGN meeting at Hiroshima 37

38 Contact Information Network Virtualization / Overlay is one of the key technologies for proceeding further in defining NwGN architecture(s) Test bed designs are crucial part of this research Net-Virtualization Research Lab Contact: nv-lab@ml.nict.go.jp Aki NAKAO, NwGN meeting at Hiroshima 38