RHEL6.x Deployment over iscsi over IPoIB Interfaces README Rev 1.0 www.mellanox.com
NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale, CA 94085 U.S.A. www.mellanox.com Tel: (408) 970-3400 Fax: (408) 970-3403 Mellanox Technologies, Ltd. Beit Mellanox PO Box 586 Yokneam 20692 Israel www.mellanox.com Tel: +972 (0)74 723 7200 Fax: +972 (0)4 959 3245 Copyright 2014. Mellanox Technologies. All Rights Reserved. Mellanox, Mellanox logo, BridgeX, ConnectX, Connect-IB, CoolBox, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, MetroX, MLNX-OS, PhyX, ScalableHPC, SwitchX, UFM, Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd. ExtendX, FabricIT, Mellanox Open Ethernet, Mellanox Virtual Modular Switch, MetroDX, TestX, Unbreakable-Link are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners. 2
Contents Rev 1.0 Contents Document Revision History... 5 1 Installing RHEL 6.x on iscsi Target... 6 2 Creating initramfs on RHEL6.x x86_64... 7 2.1 Important Notes... 8 3
Rev 1.0 Contents List of Tables Table 1: Document Revision History... 5 4
RHEL6.x Deployment over iscsi over IPoIB Interfaces README Rev 1.0 Document Revision History Table 1: Document Revision History Revision Date Description 1.0 August 7, 2014 Initial release 5
Rev 1.0 Installing RHEL 6.x on iscsi Target 1 Installing RHEL 6.x on iscsi Target To install RHEL 6.x on iscsi target: Step 1: Select "Infiniband Support". Step 2: Select "iscsi client support". Step 3: Write down root partition name. Step 4: Configure PXE server (/var/lib/tftpboot/pxelinux.cfg/default): LABEL <label> KERNEL vmilnuz APPEND initrd=initramfs-iser-2.6.32-358.el6.x86_64.img root=<root partition name> netroot=iscsi:@{iscsi-server-ip}::::{iscsi-target} iscsi_initiator={iscsi-target} ip=ib0:dhcp ifname=ib0:ff:00:00:00:00:00 rdshell Example based on RHEL6.4: LABEL rh6.4x64_iscsi MENU LABEL ^2) RHEL6.4 iscsi KERNEL RHEL6.4-x86_64-DVD1/images/pxeboot/vmlinuz APPEND initrd=rhel6.4-x86_64-dvd1/images/pxeboot/initramfs-iser-2.6.32-358. el6.x86_64.img root=/dev/sdb1 netroot=iscsi:13.7.6.30:::1:iqn.2001-04.com.sqa030-sqa030.prt9 iscsi_initiator=iqn.2001-04.com.sqa030-sqa030.prt9 ip=ib0:dhcp ifname=ib0:ff:00:00:00:00:00 rdshell # instead of changing the "init" script: APPEND initrd=rhel6.4-x86_64-dvd1/images/pxeboot/initramfs-iser-2.6.32-358. el6.x86_64.img root=/dev/sdb1 netroot=iscsi:13.7.6.30:::1:iqn.2001-04.com.sqa030-sqa030.prt9 iscsi_initiator=iqn.2001-04.com.sqa030-sqa030.prt9 ip=ib0:dhcp ifname=ib0:ff:00:00:00:00:00 bootdev=ib0 rdloaddriver=mlx4_ib,ib_ipoib,ib_iser rd_retry=120 rdshell 6
RHEL6.x Deployment over iscsi over IPoIB Interfaces README Rev 1.0 2 Creating initramfs on RHEL6.x x86_64 Step 1: Install dracut RPMs # yum install dracut-network dracut Step 2: Edit the /etc/dracut.conf file. Based on MLNX_OFED: # additional kernel modules to the default add_drivers+="ib_srp ib_iser ib_ipoib ib_sa ib_uverbs ib_core rdma_ucm ib_cm ib_ucm ib_umad ib_mad rdma_cm iw_cm ib_addr iw_c2 mlx5_ib ib_mthca ib_ipath ib_qib mlx4_ib iw_nes iw_cxgb3 iw_cxgb4 eth_ipoib cxgb3 cxgb4 mlx5_core mlx4_en mlx4_core mlx4_vnic cxgb3i libcxgbi cxgb4i scsi_transport_srp compat" Based on in-box: # additional kernel modules to the default add_drivers+="ib_srp ib_iser ib_ipoib ib_sa ib_uverbs ib_core rdma_ucm ib_cm ib_ucm ib_umad ib_mad rdma_cm iw_cm ib_addr iw_c2 ib_mthca ib_ipath ib_qib mlx4_ib iw_nes iw_cxgb3 iw_cxgb4 cxgb3 cxgb4 mlx4_en mlx4_core cxgb3i libcxgbi cxgb4i scsi_transport_srp" Step 3: Use dracut to build initramfs. # dracut initramfs-iser-`uname -r`.img `uname -r` Step 4: Open initramfs. # mkdir /tmp/initramfs # cd /tmp/initramfs # gzip -dc <../initramfs-iser-`uname -r`.img cpio -id Step 5: Make the changes below. # rm -f./etc/modprobe.d/*mlx*./etc/modprobe.d/mlnx.conf # vi init Add the text below before the line "RDRETRY=$(getarg 'rd_retry=')" modprobe mlx4_ib && echo LOADED mlx4_ib && \ modprobe mlx4_en && echo LOADED mlx4_en && \ modprobe ib_ipoib && echo LOADED ib_ipoib && \ modprobe ib_iser && echo LOADED ib_iser && \ echo Sleeping 10 sec && \ sleep 10 && \ ifup ib0 && dhclient ib0 && \ ifup ib1 && dhclient ib0 && \ ip link show Step 6: Rebuild initramfs. # find. cpio -o -c gzip -9 >../initramfs-iser-`uname -r`.img 7
Rev 1.0 Creating initramfs on RHEL6.x x86_64 2.1 Important Notes In case of MLNX_OFED installation on iscsi, partition needs to rebuild initramfs using steps 1-6 above. This should be done before the first reboot. Otherwise, the host will get the kernel panic below upon boot: "Kernel panic - not syncing: Attempted to kill init!" The same procedure can be used for RHEL6.x deployment over iscsi over Mellanox Ethernet interfaces 8