Voltaire. Fast I/O for XEN using RDMA Technologies. The Grid Interconnect Company. April 2005 Yaron Haviv, Voltaire, CTO

Transcription

1 Voltaire The Grid Interconnect Company Fast I/O for XEN using RDMA Technologies April 2005 Yaron Haviv, Voltaire, CTO

2 The Enterprise Grid Model and ization VMs need to interact efficiently with the rest of the /Grid world to be successful Storage Compute Network Environment Network/Cluster File System Middleware Layer 3-7 services Hardware LUNs, Logical Volumes VMs VLANs Hardware Disks, Arrays Servers Switches/Routers and Ports Agenda: Ways for I/O virtualization with minimal performance hit 2

3 The I/O ization Model I/O enable Environments, and complement XEN Physical Topology Servers & Blades Management Logical Topology Network Storage Servers SAN GbE GbE Remote Clients and Networks Multi-Service Switch InfiniBand Network FC,iSCSI or RDMA Storage Servers Routers SAN RAID s Network Storage Servers SAN April 13, Voltaire, Inc. 3

4 XEN Over InfiniBand Benefits IPC Network Storage Software based Network and Storage switching XEN VM0 XEN VM0 Direct HW access GbE NIC FC HCA IB HCA Slower I/O No isolation Multiple cards and fabrics Direct HW access for I/O One 10Gb/s card for Network, Storage, and IPC Scale out using clustering (IPC) Better Performance, Isolation, and TCO with InfiniBand 4

5 XEN Networking Today Linux Bridge VM0 Hypervisor NIC The problems: - Context Switches - Same Receive/Send Queue (I/O Dependencies and starvation) - Configuration/Security challenges (Multiple VLANs, routing, etc ) The Solution: - Dedicate Send/Receive Queue per NIC - Bypass Hypervisor and VM0 by using Memory Mapped I/O (separate Mem/IO Pages per VM) - Packet switching and QoS in hardware - And, still make sure the VMs are not bound to a specific Hardware 5

6 XEN Networking With Direct HW Access Init (Slow Path) VM0 Hypervisor Data Path IB HCA Slow path done in VM0 Allow direct access (DA) from Guest to data path operations (with safe DMA through mem registration) Can have a generic API (e.g. Datagram Send/Receive), loaded and used by the Ethernet front-end driver Direct access code need to be exported from VM0 to guests (like PXE/UNDI) Can be used by different hardware/nic providers (not just for IB), and allow keeping an HW neutral guest 6

7 iscsi and iser Driver Architecture iscsi-linux Open Source Project Management SCSI Low Layer In OpenIB Net IF (Ethernet/IPoIB) isns /SLP TCP iscsi TOE iser Datamover API kdapl NIC RNIC IB HCA Datamover Architecture (DA) and iser are optimal for XEN - One driver and management, support for many different cases - Centrally managed and provisioned through iscsi infrastructure (isns/slp) Same architecture apply to NICs, RNICs (Ethernet/RDMA). InfiniBand Can migrate a virtual machine from IB to Ethernet and keep on working 7

8 Path Resolution in Heterogeneous Environments GbE Router GbE Switch GbE to IB iser or TCP IPoIB FCP iser iser to FC Gateway TCP/GbE Native IB RAID IB Switch Interfaces Server iscsi TCP iser Automatic discovery of storage Automatic selection of Transport (iser or TCP) Same standard management across the entire solution - Including IP, IB, and FC storage 8

9 XEN Block Storage With Direct Access Init (Slow Path) VM0 Hypervisor IB HCA Data Path Similar design to DA Networking Need definition for direct access API (e.g. send/receive sector ) Storage front-end driver can detect and use direct access API In InfiniBand will map to separate QPs and CQs per VM, same model can apply to other HW Migration handled by iscsi recovery mechanisms (rediscover and connect) 9

10 Using XEN For Scale Out Traditional OS n VM s per host Scale out with VM s and Clustering Applications OS Hypervisor/VM0 /1 /2 Cluster Middleware Hypervisor/VM0 Hypervisor/VM0 CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU Low latency and zero copy IPC is critical to clustering Need new Front-End drivers for IPC - Socket/TOE abstraction (e.g. a BSD Scok_stream), can be used between VMs - RDMA abstraction (e.g. DAPL/IT) Scale out with XEN 10

11 IPC and Direct User Access to HW Init (Slow Path) VM0 Hypervisor Data Path IB HCA Recommend mapping to an industry standard such as DAPL - Allow existing RDMA clients such as MPI, Oracle, NFS/RDMA, iser to work transparantly - Mapped rather well to IB & Ethernet/RDMA Open issues around migration, will initially require application awareness (rediscover and reconnect on failure) Open XEN to new real-time, HPC, Media, and transactional markets 11

12 Summary InfiniBand and RDMA can solve the I/O problem TODAY - Dedicated I/O per VM - Direct access from VM to Hardware through channel architecture - Protected DMA - Same card can provide virtual network, storage and IPC (and provide the best performance in each category) Can build a single infrastructure in XEN for: - Plain Ethernet, InfiniBand and Ethernet/RDMA - Allow migration between heterogeneous I/O hardware Welcome people to participate and in an open source environment 12

13 Call for Action Establish a direct access I/O working group - Open discussion on architecture and implementation alternatives - Looking for volunteers to help code missing pieces Incorporate required changes in OpenIB (in Linux Kernel) Incorporate work into distros (SuSe, RH,..) Work with CPU vendors to optimize memory and interrupt sharing mechanisms 13