Virtualization Provisioning & Centralized Management with iscsi. RACF Brookhaven National Laboratory James Pryor CHEP 2010

Transcription

1 RACF Brookhaven National Laboratory James Pryor CHEP 2010

2 Who we are & What we support Virtualization Provisioning & Management Past Present Future This is not about VM worker nodes or VM batch processing but instead about VMs as servers & and support infrastructure.

3 We are the RACF at Brookhaven National Laboratory RHIC & US ATLAS Computing Facility Tier 0 for onsite Relativistic Heavy Ion Collider in NY, USA Tier 1 for in support of US ATLAS experiment of the LHC Part of the BNL Physics department in support of the collider experiments and not general IT support for the Lab

4 Virtualization Past Began about 3 years ago with RHEL 5.2 Workstation Not RHEL Server due to license constraints Paravirtualization with Xen Only non-critical services & servers Only US ATLAS

5 Virtualization Present Now on RHEL 5.5 Workstation Paravirtualization with Xen, physical to virtual migration with full virtualization with Xen, and small number of VMs with full virtualization with KVM Mix of non-critical, important and even critical services hosted on VMs Now both RHIC & US ATLAS Provisioning with PXE, RHEL Kickstart, mix of custom scripts, RHEL Satellite

6 Virtualization Present: Challenges Decentralized Storage for VMs spread across many host machines Manual instantiation of VMs Complexity Many layers on many host machines VM Resource Allocation: network, storage, RAM, CPU Scalability Storage # of machines Time of completion from request for VM to provisioning the VM

7 Virtualization Present: Solutions? Can be centralize & simplify while continuing to grow? Examined technologies, products, & software that might help us iscsi for storage Cobbler/Koan provisioning Puppet for configuration management Perhaps some kind of virtualization managment & administration toolset

8 ISCSI Consulted with on-site RACF storage management experts and learned about iscsi Researched products from various vendors Requirements: Simple interface for management Multiple redundant network ports Redundant controllers Cheap so we could get a pair for HA

9 Selected Nexsan SATABoy as our test platform for iscsi

10 SATABoy Technical Specifications Dimensions - Height 3U 132.5mm Power 2 x 400W load-sharing PSUs (12V / 5.1V / 3.3V / 1.8V) or 2 x 500W for later models and those fitted with SAS drives. Cooling 2 x Radial Blowers per PSU (4,200 / 4,500 rpm Max); software speed controlled. Communication Interfaces 10/100/1000 Base-T Ethernet RJ45 (two per controller, one for GUI, two for iscsi). Supports TCP/IP, HTTP, SMTP, SNMP traps, Daytime and FTP. Optional SSL for firmware xe66 and beyond. GUI HTML supported by most Internet browsers. sent via SMTP in event of failure or warning event. Trap sent via SNMP in event of failure or warning event RS232 Serial Interface DB9 (one per controller) Supports VT100. Compatible with terminal emulation software, such as Hyper Terminal and Kermit. External Data Interface Dual port, i.e. 2 host ports per controller, 4Gb/s Fiber Channel. Earlier models have 2Gb/s Fiber Channel capabilities. Option for dual port 12Gb/s SAS host interface with SATABoy2-SH. Each port uses a 26-way mini- SAS connector, suitable for use with external cabling terminated with SFF-8088 mini-sas plugs?. Each connector contains 4-lanes of SAS, with 3Gb/s capability on each lane, equating to 12Gb/s per connector.

11

12 On SATABoy 1 On SATABoy 2

13 On the VM host machine ~]# iscsiadm -m discovery -t sendtargets -p :3260,1 iqn com.nexsan:p0:sataboy2:02a75ed9 ~]# /etc/init.d/iscsi restart Stopping iscsi daemon: iscsid dead but pid file exists [ OK ] Starting iscsi daemon: [ OK ] Setting up iscsi targets: Logging in to [iface: default, target: iqn com.nexsan:p0:sataboy2:02a75ed9, portal: ,3260] Login to [iface: default, target: iqn com.nexsan:p0:sataboy2:02a75ed9, portal: ,3260]: successful [ OK ] [root@virtclus01 ~]# iscsiadm -m discovery -t sendtargets -p :3260,1 iqn com.nexsan:p0:sataboy2:02a75ec3 [root@virtclus01 ~]# /etc/init.d/iscsi restart Logging out of session [sid: 1, target: iqn com.nexsan:p0:sataboy2:02a75ed9, portal: ,3260] Logout of [sid: 1, target: iqn com.nexsan:p0:sataboy2:02a75ed9, portal: ,3260]: successful Stopping iscsi daemon: iscsid dead but pid file exists [ OK ] Starting iscsi daemon: [ OK ] Setting up iscsi targets: Logging in to [iface: default, target: iqn com.nexsan:p0:sataboy2:02a75ec3, portal: ,3260] Logging in to [iface: default, target: iqn com.nexsan:p0:sataboy2:02a75ed9, portal: ,3260] Login to [iface: default, target: iqn com.nexsan:p0:sataboy2:02a75ec3, portal: ,3260]: successful Login to [iface: default, target: iqn com.nexsan:p0:sataboy2:02a75ed9, portal: ,3260]: successful [ OK ]

14 On SATABoy 1 On SATABoy 2

15 On VM host machine ~]# fdisk -l /dev/sdc Disk /dev/sdc: 51.2 GB, bytes 64 heads, 32 sectors/track, cylinders Units = cylinders of 2048 * 512 = bytes Disk /dev/sdc doesn't contain a valid partition table [root@virtclus01 ~]# fdisk -l /dev/sdd Disk /dev/sdd: 51.2 GB, bytes 64 heads, 32 sectors/track, cylinders Units = cylinders of 2048 * 512 = bytes Disk /dev/sdd doesn't contain a valid partition table [root@virtclus01 ~]# fdisk -l Disk /dev/sdc: 51.2 GB, bytes 64 heads, 32 sectors/track, cylinders Units = cylinders of 2048 * 512 = bytes Device Boot Start End Blocks Id System /dev/sdc fd Linux raid autodetect Disk /dev/sdd: 51.2 GB, bytes 64 heads, 32 sectors/track, cylinders Units = cylinders of 2048 * 512 = bytes Device Boot Start End Blocks Id System /dev/sdd fd Linux raid autodetect

16 On VM host machine ~]# mdadm --create /dev/md0 -n 2 -l 1 /dev/sdc1 /dev/sdd1 mdadm: array /dev/md0 started. [root@virtclus01 ~]# pvcreate /dev/md0 Physical volume "/dev/md0" successfully created [root@virtclus01 ~]# vgcreate vmdisk1 /dev/md0 Volume group "vmdisk1" successfully created [root@virtclus01 ~]# vgdisplay --- Volume group --- VG Name vmdisk1 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size GB PE Size 4.00 MB Total PE Alloc PE / Size 0 / 0 Free PE / Size / GB VG UUID clqo4a-u2lr-fyyk-ooin-1hvx-cz77-ed2vda

17 On VM host machine ~]# lvcreate -L 47GB -n vmdisklv vmdisk1 ~]# lvdisplay vmdisk1 --- Logical volume --- LV Name /dev/vmdisk1/vmdisk1lv VG Name vmdisk1 LV UUID GnKFX4-YQOk-VYlx-hvem-KqTY-T2TZ-yJ9Jxi LV Write Access read/write LV Status available # open 0 LV Size GB Current LE Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253: [root@virtclus01 ~]# virt-install -b br0 -n gcebuild04 -r vcpus=2 disk \ path=/dev/vmdisk1/vmdisk1lv --os-type=linux --os-variant=rhel5.4 -m 00:aa:bb:cc:dd:ee \ -p -x ks= \ -l

18 Success! VM provisioned on centralized redundant iscsi storage.

19 Virtualization Future Ideas & Directions iscsi backed by a real SAN HA cluster Attach iscsi exported volume to VM with Linux dmmultipath Provision VMs using Cobbler/Koan and then configure the new VM with Puppet Use a virtualization management tool: RedHat Enterprise Virtualization, ConVirt, ovirt, OpenQRM, Karesansui VM live migration Looking forward to KVM with RHEL 6 KSM, qcow2 disk format, updated libvirt, libguestfs