Mellanox CloudX, Mirantis Fuel Solution Guide

Transcription

1 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0

2 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 0 Oakmead Parkway Sute 0 Sunnyvale, CA 0 U.S.A. Tel: (0) 0-00 Fax: (0) 0-0, Ltd. Bet Mellanox PO Box Yokneam Israel Tel: + (0) 0 Fax: + (0) Copyrght.. All Rghts Reserved. Mellanox, Mellanox logo, BrdgeX, ConnectX, Connect-IB, CoolBox, CORE-Drect, InfnBrdge, InfnHost, InfnScale, MetroX, MLNX-OS, PhyX, ScalableHPC, SwtchX, UFM, Vrtual Protocol Interconnect and Voltare are regstered trademarks of, Ltd. ExtendX, FabrcIT, Mellanox Open Ethernet, Mellanox Vrtual Modular Swtch, MetroDX, TestX, Unbreakable-Lnk are trademarks of, Ltd. All other trademarks are property of ther respectve owners. MLNX--

3 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0 Table of Contents Preface... Overvew... Vrtualzaton.... eswtch Capabltes and Characterstcs.... Performance Measurements... Storage Acceleraton... Networkng.... Network Types..... Admn (PXE) Network..... Storage Network..... Management Network..... Prvate Networks..... Publc Network.... Physcal Connectvty.... Network Separaton.... Lossless Fabrc (Flow-Control)... Requrements.... Hardware Requrements.... Operatng Systems... Rack Confguraton.... Compute Setup..... Fuel Node..... Controller/Compute Node..... Storage Node.... Compute Setup..... Fuel Node..... Controller/Compute Node..... Storage Node... Installaton and Confguraton...

4 Rev.0 Overvew Preface About ths Manual Ths manual s a reference archtecture and an nstallaton gude for a small sze OpenStack cloud of - compute nodes based on Mellanox nterconnect hardware and Mrants Fuel software. Audence Ths manual s ntended for IT engneers, System Archtects or any personnel who s nterested n understandng or deployng Mellanox CloudX, usng Mrants Fuel. Related Documentaton For addtonal nformaton, see the followng documents: Document Mellanox OpenStack Reference Archtecture Mellanox MLNX-OS User Manual HowTo Install Mrants Fuel OpenStack wth Mellanox Adapters Support HowTo Confgure GbE Lnk on Mellanox Adapters and Swtches Frmware - Drver Compatblty Matrx Mellanox OFED Drver Installaton and Confguraton for SR-IOV Mrants Openstack nstallaton gude HowTo Confgure SER Block Storage for OpenStack Cloud wth Mellanox ConnectX- Adapters Locaton NOTE: actve support account requred to access manual g=lnux_sw_drvers

5 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0 Overvew Mellanox CloudX s reference archtecture for the most effcent cloud nfrastructure whch makes use of open source cloud software, such as OpenStack, whle runnng on Mellanox nterconnect technology. CloudX utlzes off-the-shelf buldng blocks (servers, storage, nterconnect and software) to form flexble and cost-effectve prvate, publc, and hybrd clouds. In addton, t ncorporates vrtualzaton wth hgh-bandwdth and low-latency nterconnect solutons whle sgnfcantly reducng data center costs. Bult around the fastest nterconnect technology of 0Gb/s and Gb/s Ethernet, CloudX provdes the fastest data transfer and most effectve utlzaton of computng, storage and Flash SSD components. Based on Mellanox hgh-speed, low-latency converged fabrc, CloudX provdes sgnfcant cost reductons n CAPEX and OPEX n the followng means: Hgh VM rate per compute node Effcent CPU utlzaton due to hardware offloads Hgh throughput per server, for compute and hypervsor tasks Fast, low-latency access to storage Mrants OpenStack s one of the most progressve, flexble, open dstrbutons of OpenStack. In a sngle commercally supported package, Mrants OpenStack combnes the latest nnovatons from the open source communty wth the testng and relablty expected of enterprse software. The ntegraton of Mrants Fuel software and Mellanox Hardware generates the best soluton for cloud provder. The soluton dscussed n ths gude s based on Mrants Fuel. software (OpenStack Havana release) whch provdes the followng features: SR-IOV support on the compute nodes SER (SCSI over RDMA) block storage protocol for Cnder Fabrc speeds of up to GbE based on Mellanox SX Ethernet swtch systems

6 Rev.0 Vrtualzaton Vrtualzaton Sngle Root IO Vrtualzaton (SR-IOV) allows a sngle physcal PCIe devce to present tself as multple devces on the PCIe bus. Mellanox ConnectX - adapters are capable of exposng up to vrtual nstances called Vrtual Functons (VFs). These VFs can then be provsoned separately. Each VF can be vewed as an addtonal devce assocated wth a Physcal Functon (PF). The VF shares the same resources wth the PF, and ts number of ports equals those of the PF. SR-IOV s commonly used n conjuncton wth an SR-IOV enabled hypervsor to provde vrtual machnes (VMs) wth drect hardware access to network resources, thereby mprovng performance. Mellanox ConnectX- adapters equpped wth an onboard embedded swtch (eswtch) are capable of performng Layer- swtchng for the dfferent VMs runnng on the server. Usng the eswtch yelds even hgher performance levels, as well as mproves securty and solaton. The nstallaton s capable of handlng Mellanox NIC cards, t updates the correct frmware verson whch ncorporates SR-IOV enablement and defnes VFs. Each spawned VM provsoned wth one VF per network attached. The soluton supports up to VMs on a sngle compute node connected to sngle network or VMs connected to networks or any other combnaton whch sums to networks n total. SR-IOV support for OpenStack s under development. Securty groups are not supported wth SR-IOV. If the setup s based on Mellanox OEM NICs, make sure to have a compatble frmware verson to OFED verson (or later). Make sure that ths frmware verson supports SR-IOV (clck here for addtonal nformaton). Fgure : eswtch Archtecture

7 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0. eswtch Capabltes and Characterstcs The man capabltes and characterstcs of eswtch are lsted n the followng: Vrtual swtchng: Creatng multple logcal vrtualzed networks. The eswtch offload engnes handle all networkng operatons up to the VM, thereby dramatcally reducng software overheads and costs. Performance: Swtchng s handled by hardware as opposed to other applcatons that use a software-based swtch. Ths enhances performance by reducng CPU overhead. Securty: The eswtch enables network solaton (usng VLANs) and ant-mac spoofng. Montorng: Port counters are supported.. Performance Measurements Many data center applcatons beneft from low-latency network communcaton whle others requre determnstc latency. Usng regular TCP connectvty between VMs can create hgh latency and unpredctable delay behavor. Fgure shows the dramatc dfference (X mprovement) delvered by SR-IOV connectvty runnng RDMA compared to para-vrtualzed vnic runnng a TCP stream. Usng the drect connecton of SR-IOV and ConnectX-, the hardware elmnates software processng whch delays packet movement. Ths result n consstent low-latency that allows applcaton software to rely on determnstc packet transfer tmes. Fgure : Latency Comparson

8 Rev.0 Storage Acceleraton Storage Acceleraton Data centers rely on communcaton between compute and storage nodes as compute servers read and wrte data from storage servers constantly. To maxmze the server s applcaton performance, communcaton between the compute and storage nodes must have the lowest possble latency, hghest possble bandwdth, and lowest CPU utlzaton. Fgure : OpenStack Based IaaS Cloud POD Deployment Example Storage applcatons, relyng on SCSI over TCP communcatons protocol stack, contnuously nterrupt the processor to perform basc data movement tasks (packet sequence and relablty tests, re-orderng, acknowledgements, block level translatons, memory buffer copyng, etc). Ths causes data center applcatons that rely heavly on storage communcaton to suffer from reduced CPU effcency, as the processor s busy sendng data to and from the storage servers rather than performng applcaton processng. The data path for applcatons and system processes must wat n lne wth protocols such as TCP, UDP, NFS, and SCSI for ther turn to use the CPU. Ths not only slows down the network, but also uses system resources that could otherwse have been used for executng applcatons faster. Mellanox OpenStack soluton extends the Cnder project by addng SCSI runnng over RDMA (SER). Leveragng RDMA Mellanox OpenStack delvers X better data throughput (for example, ncreasng from GB/s to GB/s) and whle smultaneously reducng CPU utlzaton by up to 0% (see Fgure ). Mellanox ConnectX - adapters bypass the operatng system and CPU by usng RDMA, allowng much more effcent data movement. SER capabltes are used to accelerate hypervsor traffc, ncludng storage access, VM mgraton, and data and VM replcaton. The use of RDMA shfts data movement processng to the Mellanox ConnectX- hardware, whch provdes zero-copy message transfers for SCSI packets to the applcaton, producng sgnfcantly faster performance, lower network latency, lower access tme, and lower CPU overhead. SER can provde X faster performance than tradtonal TCP/IP based SCSI. The SER protocol unfes the software development efforts of both Ethernet and InfnBand communtes, and reduces the number of storage protocols a user must learn and mantan.

9 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0 RDMA bypass allows the applcaton data path to effectvely skp to the front of the lne. Data s provded drectly to the applcaton mmedately upon recept wthout beng subject to varous delays due to CPU load-dependent software queues. Ths has three effects: The latency of transactons s ncredbly low; Because there s no contenton for resources, the latency s determnstc, whch s essental for offerng end users a guaranteed SLA; Bypassng the OS, usng RDMA results n sgnfcant savngs n CPU cycles. Wth a more effcent system n place, those saved CPU cycles can be used to accelerate applcaton performance. In Fgure t s clear that by performng hardware offload of the data transfers usng the SER protocol, the full capacty of the lnk s utlzed to the maxmum of the PCIe lmt. To summarze, network performance s a sgnfcant element n the overall delvery of data center servces and benefts from hgh speed nterconnects. Unfortunately, the hgh CPU overhead assocated wth tradtonal storage adapters prevents systems from takng full advantage of these hgh-speed nterconnects. The SER protocol uses RDMA to shft data movement tasks to the network adapter and thus frees up CPU cycles that would otherwse be consumed executng tradtonal TCP and SCSI protocols. Hence, usng RDMA-based fast nterconnects sgnfcantly ncrease data center applcaton performance levels. Fgure : RDMA Acceleraton

10 Rev.0 Networkng Networkng In ths soluton, we defne the followng node functons: Fuel node (master) Compute nodes Controllers (and network) node Storage node (+ JBODs) The followng fve networks are requred for ths soluton: Publc network Admn (PXE) network Storage network Management network Prvate network Besdes the Fuel node that s connected only to the Publc and Admn (PXE) networks, all other nodes are connected to all fve networks. Although not all nodes may be requred to connect to all networks, ths s done by Fuel desgn. Fgure - Soluton Networkng Prvate, Management Networks Storage Network Fuel Master Compute Storage Controller Node Admn (PXE) Network Publc Network Frewall Internet Publc Network Prvate, Management Networks Storage Network Admn (PXE) Network

11 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0. Network Types.. Admn (PXE) Network The Admn (PXE) network s used for the cloud servers PXE boot and OpenStack nstallaton. It uses the GbE port on the servers. Fgure : PXE Network Fuel Master Compute Storage Controller Node Admn (PXE) Network.. Storage Network The Storage network s used for tenant storage traffc. The storage network s connected va the SX (0/GbE). It s recommended to connect all compute and storage nodes va port n the ConnectX - Pro network adapter. The SER protocol runs between the hypervsors and the Storage node over the 0/GbE storage network. The VLAN used for the storage network s confgured by the Fuel UI. Fgure : Storage Network Storage Network Fuel Master Compute Storage Controller Node.. Management Network The Management network s an nternal network whch medates among between the controller, storage, and compute nodes. It s connected wth the SX (0/GbE swtch). It s recommended to connect the relevant servers usng port n the ConnectX- Pro network adapter. The VLAN used for the management network s confgured by the Fuel UI.

12 Rev.0 Networkng Fgure : Management Network Management Network Fuel Master Compute Storage Controller Node.. Prvate Networks The prvate networks are used for communcaton among the tenant VMs. Each tenant may have several networks. If connectvty s requred between networks owned by the same tenant, the Network node does the routng. It s recommended to connect the relevant servers through port n the ConnectX- Pro network adapter (same port as the Management network). The VLAN used for the management network s confgured va the Fuel UI. Fuel. s based on OpenStack Havana whch does not support more than one network technology. Ths means that all the prvate networks n the OpenStack deployment should use Mellanox Neutron agent whch s based on VLANs assgned to VFs. The VLAN range used for prvate networks s confgured by the Fuel UI. Note: Allocate number of VLANs accordng to the number of prvate networks to be used. Fgure : Prvate Network Tenant A, net Tenant A, net Tenant B Tenant C Controller & Network Node VMs VMs VMs VMs.. Publc Network The publc network enables external connectvty to all nodes (e.g. Internet). The publc network runs on the GbE ports of each server. Ths network s also used to access the dfferent OpenStack APIs.

13 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0 The publc network range s splt nto two parts: Publc range: Allows external connectvty to the compute hypervsors and all other hosts Floatng IP range: Enables VMs to communcate wth the outsde world and s a subset of addresses wthn the publc network (va the controller node) Another use for publc access for hypervsors or node OS mght be necessary (e.g. ssh access). However, the cloud owner must decde how to allow external connectvty to cloud servers, or cloud servers access to nternet (ether PXE or the Publc network may be used). Fgure : Publc Network Fuel Master Compute Storage Controller Node Publc Network Frewall Internet The floatng IP uses the prvate network to pass traffc from the VMs to the Network node and then the Publc network to pass traffc from the Network node to the Internet.

14 Rev.0 Networkng Fgure : Floatng-IP Network Prvate Networks NAT Operaton Fuel Master Compute Storage Controller Node Publc Network Frewall Internet. Physcal Connectvty The fve networks dscussed above can be connected usng - GbE swtches and - SX swtches (0/GbE swtches) dependng on the number of servers and the sze of the swtches. In the case of up to nodes ( compute nodes), t s possble to use only one SX swtch ( ports). In case of up to nodes ( compute nodes), two SX are necessary (one for the storage network and one for prvate/management networks. In the ConnectX- Pro adapter, the lower port has the lower MAC address. When usng Fuel UI to assgn networks t s recommended to assgn the Management and Prvate networks to port (whch has the lower MAC address) and the Storage network to the port wth the hgher MAC address. For example:

15 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0 Fgure : Network Confguaton - Example. Network Separaton VLANs should be confgured on the swtches to allow network separaton. The VLANs to be confgured on the swtches should be algned wth the VLAN range confgured n Fuel for the Management, Prvate and Storage networks.. Lossless Fabrc (Flow-Control) RDMA requres lossless fabrc. To acheve that, flow control (global pause) should be enabled on all ports runnng RDMA and connected to the SX Ethernet swtch. Ths should be acheved by confgurng global pause across all network hardware components.

16 Rev.0 Requrements Requrements Mrants Fuel defnes the followng node functons: Fuel node Compute nodes Controller node Storage nodes. Hardware Requrements Table lsts the mnmum requrements of hardware components: Table : Hardware Requrements Component Quantty Descrpton Fuel node (master) Intel server, Gb Eth Ports, > cores CPU, >=GB RAM, 0. TB SATA HDD Compute nodes - Intel PCI-Ex Gen- server, at least one x PCI-Ex slot, Gb Eth Ports, > cores CPU x, >=GB RAM RAM, 0. TB SATA HDD, SRIOV support n BIOS. ConnectX - PRO VPI Dual Port network adapter. P/N MCXA-FCCT Controller node Intel PCI-Ex Gen server, at least one x PCI-Ex slot, > x Gb Eth Ports, > cores CPU x, >= GB RAM, TB SAS HDD, SRIOV support n BIOS Storage node (Clck here) ConnectX- PRO VPI Dual Port network adapter. P/N: MCXA-FCCT Storage, Management Swtch Publc, Admn (PXE) swtch or Mellanox SX 0/GbE ports or Gb swtch (any swtch) Gb/s cables per server FDR InfnBand/GbE copper cables up to m. P/N: MC0-XXX. Gb/s cables per server. Operatng Systems All servers operatng systems are nstalled wth CentOS. (Mrants dstro) va Fuel servers. The VM should have the approprate drver to support SR-IOV mode over Mellanox ConnectX-.

17 Extreme Networks Summt s Extreme Networks Summt s Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0 Rack Confguraton. Compute Setup Ths secton supples a rack recommendaton desgn for basc cloud setup up to nodes ( compute nodes). Fgure : Rack Confguraton PORT 0 PORT Publc Swtch PORT 0 PORT Admn (PXE) Swtch Compute Node... Compute Node Controller Node Fuel Node Management, Prvate Swtch Storage Swtch Storage Server JBOD JBOD

18 Fuel Extreme Networks Summt s Extreme Networks Summt s Extreme Networks Summt s Extreme Networks Summt s 0 0 Extreme Networks Summt s Extreme Networks Summt s Rev.0 Rack Confguraton.. Fuel Node Fgure : Fuel Node PORT 0 PORT PORT 0 PORT Publc Swtch Admn (PXE) Swtch eth0 eth Node.. Controller/Compute Node Fgure : Controller/Compute Node PORT 0 PORT PORT 0 PORT Publc Swtch Admn (PXE) Swtch eth (port) eth0 eth eth (port) Compute/Controller Node Management, Prvate Swtch Storage Swtch.. Storage Node Fgure : Storage Node PORT 0 PORT PORT 0 PORT Publc Swtch Admn (PXE) Swtch JBOD Management, Prvate Swtch Storage Swtch SAS eth (port) eth (port) eth0 eth Storage Server JBOD SAS

19 Extreme Networks Summt s Extreme Networks Summt s Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0. Compute Setup In setups wth up to nodes ( compute nodes) t s possble to reduce the number of swtches. Use only one GbE swtch for all GbE networks (splt va VLAN) Use only one 0/GbE swtch (SX) for both Prvate, Management and Storage networks (use VLANs to splt the networks) Fgure : Rack Confguraton PORT 0 PORT Publc Swtch PORT 0 PORT Admn (PXE) Swtch Compute Node... Compute Node Controller Node Fuel Node Management, Prvate, Storage Swtch Storage Server JBOD JBOD

20 Fuel Extreme Networks Summt s Extreme Networks Summt s Extreme Networks Summt s Extreme Networks Summt s 0 0 Extreme Networks Summt s Extreme Networks Summt s Rev.0 Rack Confguraton.. Fuel Node Fgure : Fuel Node PORT 0 PORT PORT 0 PORT Publc Swtch Admn (PXE) Swtch eth0 eth Node.. Controller/Compute Node Fgure : Controller Node PORT 0 PORT PORT 0 PORT Publc Swtch Admn (PXE) Swtch eth (port) eth0 eth eth (port) Controller/ Compute Node Management, Prvate, Storage Swtch.. Storage Node Fgure : Storage Node PORT 0 PORT PORT 0 PORT Publc Swtch Admn (PXE) Swtch JBOD Management, Prvate, Storage Swtch SAS eth (port) eth (port) eth0 eth Storage Server JBOD SAS

21 Mellanox CloudX, Mrants Fuel Soluton Gude Rev.0 Installaton and Confguraton For cloud nstallaton and confguraton clck here. For storage node nstallaton and confguraton clck here.