Linux Diskless iscsi Boot HowTo ( V1.0)

Linux Diskless iscsi Boot HowTo ( V1.0) This document describes using the Cisco Linux 3.4.2 iscsi initiator for diskless booting of Red Hat Linux 9 (RH9). EqualLogic has enhanced the initiator to provide boot support using the open-source PXELINUX network bootloader and has provided a utility for creating the diskless hosts boot disk and kernel boot images. Before continuing you should be familiar with: Installing Red Hat Linux 9 Setting up and configuring a dhcp and tftp server PXELINUX (http://syslinux.zytor.com/pxe.php) Installation and usage of the Cisco Linux iscsi Initiator (http://sourceforge.net/projects/linux-iscsi) High Level Overview of the Boot Process 1) The PC boots using the NIC s PXE support. A dhcp request is made. The dhcp response returns the hosts IP addressing info, the IP address of the tftp sever and the filename to load (pxelinux.0). 2) PXE loads via tftp the pxelinux.0 network bootloader image and executes it. PXELINUX gets, via tftp, a configuration file for this particular host the file name requested is an ASCII hex representation of the hosts IP (Ex: 192.168.0.106 = C0A8006A). The configuration file contains parameters telling what the root disk and partition will be; the iscsi target IP address and iscsi name; the hosts IP address information and Linux kernel and initrd file file names. 3) PXELINUX loads via tftp the specified kernel image and executes it passing it the configuration file parameters as boot command line arguments. The initrd (ram disk) is loaded. 4) The /linuxrc file is executed. This loads the NIC, SCSI and iscsi drivers and passes control to /bin/init. Init (from the Cisco Linux Initiator kit) instructs the iscsi driver to bring up basic networking and establish an iscsi session. Init then maps the resulting disk in as the system disk. The remainder of a normal Linux boot takes place. EqualLogic, Inc Page 1 1/25/2008

Restrictions: Red Hat Linux 9 with kernel 2.4.20-6 or later. The 2.6 kernel is not supported. Only one (1) NIC (eth0) on the iscsi diskless host is supported. Boot (system) disk must be a single disk. The RH9 installation can automatically partition the installation disk for you,, resulting in 3 partitions (/, /boot, swap) with ext3 file systems. You could also choose to manually partition the disk and use 2 partitions (/ and swap). Var (/var) must be on the same partition as root (/). i.e. it is not a mounted partition. File system types: ext2, ext3, reiserfs Chap cannot be use to authenticate the boot disk only an IP address ACL and/or initiator name can be used. The dhcp server must be configured to use static addresses if IP address ACL s (recommended) are used on the iscsi volumes. Basic Installation Requirements: dhcp and tftp servers. These servers can be located on the same system. Example dhcpd.conf (for RH9), tftp layout and PXELINUX configuration files are described in the appendices. EqualLogic PeerStorage Array Obtain from EqualLogic the Cisco Linux 3.4.2 iscsi initiator with PXELINUX support (linux-iscsi-3.4.2.pxe_iscsi_boot-1.0.tgz) A PC with a local hard disk and NIC. RH9 and the Cisco initiator will be installed on this system. This will become the master image PC from which the iscsi boot disk for the diskless host(s) will be cloned. A working iscsi connection between the master image PC and the PeerStorage Array. A PC with a Linux supported NIC with PXE support. This system will be the diskless host. For testing purposes this could be the same system as the master image PC. EqualLogic, Inc Page 2 1/25/2008

Gathering Information for the Installation: Prior to starting installation gather the following information: In the example information here, the diskless host system and the installation master are the same system Diskless Host hostname NIC driver client6 e100 IP Address 192.168.0.106 Netmask 255.255.255.0 Gateway IP address 192.168.0.1 NIC MAC address 0:C0:9F:08:D0:59 Target Group address 192.168.0.13 Targetname iqn.2001-05.com.equallogic:6-8a0900-3d1810001- c7f00086b0c40599-client6 EqualLogic, Inc Page 3 1/25/2008

System that will be the installation master hostname client6 IP Address 192.168.0.106 Netmask 255.255.255.0 Gateway address 192.168.0.1 MAC address Targetname Path to iscsi Kit 0:C0:9F:08:D0:59 iqn.2001-05.com.equallogic:6-8a0900-3d1810001- c7f00086b0c40599-client6 /usr/src/ linux-iscsi-3.4.2.pxe_iscsi_boot-1.0 EqualLogic, Inc Page 4 1/25/2008

dhcp and tftp Server Configuration and Testing: dhcp Server: Appendix A provides a RH9 /etc/dhcp.conf config file. A copy of this file is provided in the PXElinux directory of the initiator kit. Key points to the configuration are: Host addressing is static and associated with a specific MAC address The filename for the tftp request must be pxelinux.0 tftp Server: Appendix B gives an example directory listing of a /tftpboot directory setup for PXELINUX booting. Testing: It is recommended that you do not proceed with iscsi diskless booting until this testing is successfully accomplished. This testing leaves iscsi out of the picture and verifies that dhcp, tftp and PXE boot all works. Provided in the PXElinux directory of the initiator kit are the files; testkern; testrd.img and test.pxelinux.cfg. These files are a 2.4.20-6 RH9 kernel, an initrd image and an associated PXELINUX config file. To test your dhcp and tftp servers and a PC with a PXE enabled NIC do the following: 1) Obtain the MAC address of the PC s NIC that will perform the PXE boot. 2) Choose an IP address for the PC. 3) Using the MAC and IP address create an appropriate PXE booting entry on the dhcp server. Be sure the dhcp server recognizes the change. See the example dhcpd conf file in Appendix A. 4) Place a copy of testkern and testrd.img in /tftpboot on the tftp server. EqualLogic, Inc Page 5 1/25/2008

5) In the PXElinux directory of the initiator kit is the utility gethostip (comes with the PXELINUX distribution). This utility takes an IP address in standard dot format as an argument and returns an ASCII hex representation of that address. The IP address you use is the above from step #2, that which dhcp will assign to the PC being PXE booted. For example: #./gethostip -x 192.168.0.106 C0A8006A You must use the result (Ex: C0A8006A) as the file name for the PXELINUX host config file. Copy the supplied test.pxelinux.cfg file to /tftpboot/pxelinux.cfg/ on the tftp server using the hex address name for the file. See Appendix B for a listing of the /tftpboot directory. 6) Do a PXE boot of the PC: a) PXE will do a dhcp request and be returned an IP address and a file name (pxelinux.0) to load via tftp and then execute. b) pxelinux.o is loaded and run. Its requests its config file from the tftp server the file name being the ASCII hex representation of its IP address. The contents of this file tell PXELINUX the file that it is to load via tftp (DEFAULT parameter) and the arguments to pass to the Linux kernel (APPEND parameter). c) The Linux kernel file (testkern) is loaded and executed. Linux then loads the specified initrd image (testrd.img). When Linux finishes booting you will be left at a sash shell prompt >. You can do a very limited set of commands at this point (enter help for a list) (see: http://qslinux.org/docs/man/sash1.html). 7) If you have successfully reached this point you have tested the correct operation of the dhcp and tftp servers and that the PC can do a PXE boot. You are ready to proceed with configuring for Linux iscsi diskless boot. Troubleshoot as necessary until this works. Watching the log files of the dhcp and tftp servers (use tail f /var/log/messages on Linux) provides valuable information see Appendix D for an example. EqualLogic, Inc Page 6 1/25/2008

PeerStorage Array Configuration: Install and configure as normal, no special requirements Creating the iscsi boot disk: 1) Install the master image PC with Red Hat Linux 9 as you normally would. This installation will be used to create the iscsi boot disk. Ideally, but not required, this should be an identical system to the diskless hosts this will prevent Linux from going through a reconfiguration if it boots on a different hardware configuration. You must install RH9 to a single disk. Either 2 partitions (/ and swap) or 3 partitions (/, /boot, swap) are recommended. Using a journaling file system (ext3) is recommended. The networking must be configured to use dhcp. Insure the dhcp server is setup to support this system. Assigning the hostname via dhcp is optional but is recommended. After the RH9 installation is complete reboot as you normally would. Complete any additional software installations and tailoring of the system. 2) Determine how large the iscsi boot disk needs to be to contain a clone of the Linux install just completed. The simple answer is the same size as the local disk you installed to. The sfdisk -s um command will tell you this size. The iscsi disk can certainly be larger, however the cloning process will not take advantage of the extra space. If the partitions created during the RH9 install do not use the entire local disk than the iscsi disk could also be sized appropriately smaller. Use the sfdisk -l um command to determine how much space each partition uses. Regardless, making the iscsi boot disk too small will cause the cloning process to fail better an extra bit bigger than not. 3) On the PeerStorage Array, create a volume of the appropriate size for the iscsi boot disk. Assign an IP address ACL to the volume. At this point, use the IP address of the master image PC. EqualLogic, Inc Page 7 1/25/2008

4) Install the Cisco Linux 3.4.2 iscsi initiator with PXELINUX support as would normally be done. Establish an iscsi connection to the volume created above. Only connect to one volume at this time. Use the iscsi-ls l command to verify the connection was correctly established. 5) In the output of that iscsi-ls l command will be the TARGET NAME. Record this name in the other table of information being collected. 6) In the PXElinux directory of the initiator kit is the mkclone.sh utility. The utility will be used to duplicate (clone) the disk Red Hat Linux 9 was installed on to the iscsi disk this will be the iscsi boot disk. Associated kernel and initrd images for the diskless iscsi host will also be created. The mkclone.sh utility will prompt for information required to create the iscsi boot disk Run the mkclone.sh now. See Appendix E for a sample session. Depending upon how full each partition is and how last the network connection is the cloning process will take about 8 minutes to complete (1000Mbps, 2GB+ in root partition). When mkclone.sh completes its will display the names of the kernel and initrd files it created and also created a template of the PXELINUX config file. 7) At this point the iscsi boot disk is created and the initrd is created. The mkclone.sh utility displayed the names of the kernel and initrd images and an example PXELINUX config file. The iscsi boot disk is unique to each diskless host. The disk cannot be shared between different diskless hosts. The kernel image is not special to iscsi diskless booting. Multiple iscsi diskless hosts could use a single image of this kernel. It is recommended that you do this so as to keep the contents of /tftpboot more manageable. The initrd image is unique only in the NIC driver that it was configured to use. Any iscsi diskless host that requires the same NIC driver could use that initrd image. It is recommended that you make use of this so as to keep the contents of /tftpboot more manageable. EqualLogic, Inc Page 8 1/25/2008

8) Put the kernel image specified by mkclone.sh in /tftpboot of the tftp server, for example: scp /boot/vmlinuz-2.4.20-6 <tftp server>:/tftpboot/ Put the initrd image specified by mkclone.sh in /tftpboot of the tftp server, for example: scp /boot/iscsird-e100.img <tftp server>:/tftpboot/ 9) The mkclone.sh utility created a template PXELINUX host config file to match the iscsi boot disk, kernel and initrd images (/boot/example_pxelinux.config). Using the diskless host s information previously gathered, edit this template file and enter the diskless host s MAC address, IP address, netmask and gateway address. See Appendix C for an example. The required config file name is a hex ASCII representation of the iscsi diskless host s IP address. In the PXElinux directory of the initiator kit is the utility gethostip (comes with the PXELINUX distribution). This utility takes an IP address in standard dot format as an argument and returns an hex ASCII representation of that address. For example: #./gethostip -x 192.168.0.106 C0A8006A Put a copy of the edited hosts config file in /tftpboot/pxelinux.cfg/ of the tftp server and rename it to be the hex ASCII name created above. For example: scp /boot/example_pxelinux.config <tftp server>:/tftpboot/pxelinux.cfg/c0a8006a 10) Using the iscsi diskless host s MAC and IP address create an appropriate host PXE booting entry on the dhcp server. Be sure the dhcp server recognizes the change. See Appendix A for an example. 11) On the PeerStorage Array, verify the Access Control List for the host s iscsi boot disk allows it access. It is recommended that an IP address ACL be used. 12) Perform a PXE boot of the iscsi diskess host. 13) Repeat the process for each iscsi diskless boot host. EqualLogic, Inc Page 9 1/25/2008

Troubleshooting: In the event that an iscsi diskless host fails to boot try one or more of these suggestions to diagnose and correct the problem: 1) Use the dhcp/tftp testing procedure described earlier to verify the configuration and operation of these servers and the diskless host system s ability to do a PXE boot. This testing eliminates iscsi from the picture. 2) Check the BIOS boot sequence settings to verify PXE boot is there and precedes booting a local disk. 3) Some NICs do not have PXE boot enabled by default. Running a NIC vender s utility to enable PXE boot or BIOS setting changes may be required. 4) On the PeerStorage Array, verify the correct Access Control List settings on the volume to allow the diskless host access. 5) Verify the correct NIC driver was specified when mkclone.sh prompted for it. 6) If there is more than one (1) NIC in the iscsi diskless host verify that Linux will only enable eth0. On the host s iscsi boot disk the appropriate ifcfg-eth* files in /etc/sysconfig/network-scripts/ should have ONBOOT=no. 7) If the diskless host boots and can not establish an iscsi session verify the correct syntax and parameter arguments in that hosts PXELINUX config file in the /tftpboot/pxelinux.cfg/ directory. The APPEND line in the config file must be all one line with no breaks and must not exceed 255 characters in length. EqualLogic, Inc Page 10 1/25/2008

Appendix: A Example RH9 dhcp.conf deny unknown-clients; not authoritative; allow bootp; allow booting; option ip-forwarding false; option mask-supplier false; ddns-update-style none; default-lease-time 21600; max-lease-time 43200; option subnet-mask 255.255.255.0; option broadcast-address 192.168.0.255; option routers 168.168.0.1; option domain-name-servers 192.168.0.1; option domain-name "dept.somecompany.com"; # # PXE booting config # subnet 192.168.0.0 netmask 255.255.255.0 { # want the tftp server to be at a fixed address group { next-server 192.168.0.1; filename "pxelinux.0"; } } host client6 { hardware ethernet 00:C0:9F:08:D0:59; fixed-address 192.168.0.106; option host-name "client6"; } EqualLogic, Inc Page 11 1/25/2008

Appendix: B Example /tftpboot layout # ls -lr /tftpboot /tftpboot: total 1752 -rw-r--r-- 1 root root 1122363 Mar 20 15:11 vmlinuz-2.4.20-6 -rw-r--r-- 1 root root 641036 Mar 20 15:08 iscsird-e100.img -rw-r--r-- 1 root root 11720 Mar 3 13:47 pxelinux.0 drwxr-xr-x 2 root root 4096 Mar 22 15:09 pxelinux.cfg -rw-r--r-- 1 root root 1122363 Mar 23 10:41 testkern -rw-r--r-- 1 root root 882297 Mar 23 10:42 testrd.img /tftpboot/pxelinux.cfg: total 4 -rw-r--r-- 1 root root 224 Mar 18 20:03 C0A8006A EqualLogic, Inc Page 12 1/25/2008

Appendix: C Example PXELINUX config files (/tftpboot/pxelinux.cfg/) Example template config file created by mkclone.sh: DEFAULT vmlinuz-2.4.20-6 APPEND iscsird-e100.img root=/dev/inbpdisk2 RP=2 MA=<boot host MAC addr> TA=172.17.5.13 TN=iqn.2001-05.com.equallogic:6-8a0900-27c810001-a6200086b724062d-client6 CA=<client IP addr> CG=<client gateway addr> CN=<client netmask> Description of parameters (NOTE: case sensitive) DEFAULT : specifes the file name in /tftpboot to be loaded as the kernel image initrd= : specifies the file name in /tftpboot to be loaded as the initrd image root= : specifies the special device partition for the root (/) file system. Do not change this from the names given by mkclone.sh as it matches the layout of the iscsi boot disk that was created. The device name (inbpdisk#) is unique to the iscsi boot support and is mapped to the normal scsi disk device name (ex: sda2). RP= :specifies the root file system partition number. Do not change this from that given by mkclone.sh as it matches the layout of the iscsi boot disk that was created. MA= :specifies the MAC address of the NIC on the iscsi diskless host that will be used for booting. TA= TN= :specifies the Group IP address of the EqualLogic PeerStorage Array where the iscsi boot disk is located. :specifies the target name of the iscsi boot disk. CA= :specifies the iscsi diskless host s (client) IP address of the NIC used to boot from. The same NIC that the MAC address was obtained from. CG= :specifies the iscsi diskless host s (client) gateway address. CN= :specifies the iscsi diskless host s (client) netmask. EqualLogic, Inc Page 13 1/25/2008

Example for an iscsi diskless boot host: DEFAULT vmlinuz-2.4.20-6 APPEND initrd=iscsird-e100.img root=/dev/inbpdisk2 RP=2 MA=00:C0:9F:08:D0:59 TA=172.17.5.13 TN=iqn.2001-05.com.equallogic:6-8a0900-3d1810001-c7f00086b0c40599-client6 CA=172.17.5.116 CG=172.17.5.1 CN=255.255.255.0 Config used for testing dhcp/tftp servers and PXE boot: DEFAULT testkern APPEND initrd=testrd.img root=/dev/ram init=/bin/sash EqualLogic, Inc Page 14 1/25/2008

Appendix: D Example dhcp and tftp server log file entries on RH9: Mar 23 12:31:12 eqlsrv dhcpd: DHCPDISCOVER from 00:c0:9f:08:d0:59 via eth0 Mar 23 12:31:12 eqlsrv dhcpd: DHCPOFFER on 192.168.0.106 to 00:c0:9f:08:d0:59 via eth0 Mar 23 12:31:13 eqlsrv dhcpd: DHCPREQUEST for 192.168.0.106 (192.168.0.1) from 00:c 0:9f:08:d0:59 via eth0 Mar 23 12:31:13 eqlsrv dhcpd: DHCPACK on 192.168.0.106 to 00:c0:9f:08:d0:59 via eth0 Mar 23 17:31:13 eqlsrv in.tftpd[27383]: RRQ from 192.169.0.106 filename pxelinux.0 Mar 23 17:31:13 eqlsrv in.tftpd[27383]: tftp: client does not accept options Mar 23 17:31:13 eqlsrv in.tftpd[27384]: RRQ from 192.168.0.106 filename pxelinux.0 Mar 23 17:31:13 eqlsrv in.tftpd[27385]: RRQ from 192.168.0.106 filename pxelinux.cf g/01-00-c0-9f-08-d0-59 Mar 23 17:31:13 eqlsrv in.tftpd[27385]: sending NAK (1, File not found) to 192.168.0.106 Mar 23 17:31:13 eqlsrv in.tftpd[27386]: RRQ from 192.168.0.106 filename pxelinux.cfg/c0a80006 Mar 23 17:31:13 eqlsrv in.tftpd[27387]: RRQ from 192.168.0.106 filename testkern Mar 23 17:31:14 eqlsrv in.tftpd[27388]: RRQ from 192.168.0.106 filename testrd.img EqualLogic, Inc Page 15 1/25/2008

Appendix: E Example of running mkclone.sh [root@client6 PXElinux]#./mkclone.sh Stopping iscsi: sync umount sync iscsid iscsi clearing /var/lib/iscsi/bindings file... Starting iscsi: iscsi iscsid fsck/mount enter source disk (Ex: /dev/hda): /dev/hda enter destination disk (Ex: /dev/sda): /dev/sda enter full path to Cisco Linux iscsi kit: /usr/src/linux-iscsi-3.4.2.pxe_iscsi_boot-1.0 enter the NIC driver the client requires (Ex: e100): e100 will iscsi diskless host get its hostname via dhcp (y/n)? y -------- CLONING PROCESS WILL USE THESE PARAMETERS -------- using source disk: hda using destination disk: sda using iscsi kit path: /usr/src/linux-iscsi-3.4.2.pxe_iscsi_boot-1.0 iscsi Target IP Address: 172.17.5.13 iscsi Target Name: iqn.2001-05.com.equallogic:6-8a0900-27c810001-a6200086 b724062d-clinet6a diskless system NIC driver: e100 diskless system hostname: set via dhcp ***WARNING*** existing disk data on destination disk sda will be destroyed! ----------------------------------------------------------- EqualLogic, Inc Page 16 1/25/2008

Do you want to use above parameters and continue (y/n)? y zeroing destination disk: /dev/sda partitioning destination disk: /dev/sda creating ext3 file system on /dev/sda2 creating ext3 file system on /dev/sda1 initializing swap on /dev/sda3 copying from: /dev/hda2 (/) to: /dev/sda2..^[... copying from: /dev/hda1 (/boot) to: /dev/sda1 Completed creating a copy of hda on sda Creating iscsi boot initrd... A kernel image located in /boot will be used for the iscsi boot Enter name of kernel image to boot [vmlinuz-2.4.20-6]: ------ WILL USE THESE PARAMETERS TO CREATE iscsi initrd ------- root file system type: ext3 kernel image: /boot/vmlinuz-2.4.20-6 intrd image: /boot/initrd-2.4.20-6.img iscsi kit: /usr/src/linux-iscsi-3.4.2.pxe_iscsi_boot-1.0 NIC DRIVER: /lib/modules/2.4.20-6/kernel/drivers/net/e100/e100.o --------------------------------------------------------------- EqualLogic, Inc Page 17 1/25/2008

Do you want to use above parameters and continue (y/n)? y initrd image saved as: /boot/iscsird-e100.img matching kernel image: /boot/vmlinuz-2.4.20-6 NOTE: This kernel and initrd image should be placed in /tftpboot of the tftp server Done creating iscsi boot initrd image... ---- EXAMPLE pxelinux config file: /boot/example_pxelinux.config ---- DEFAULT vmlinuz-2.4.20-6 APPEND iscsird-e100.img root=/dev/inbpdisk2 RP=2 MA=<boot host MAC addr> TA=172.17.5.13 TN=iqn.2001-05.com.equallogic:6-8a0900-27c810001-a62000 86b724062d-client6 CA=<client IP addr> CG=<client gateway addr> CN=<client netmask> ---------------------------------------------------------------------------------------- Done... EqualLogic, Inc Page 18 1/25/2008