Hernán J. Larrea NetApp Data Ontap Simulator Cookbook HOW TO BUILD YOUR OWN VIRTUAL, ALL FUNCTIONAL STORAGE SIMULATOR, WITH UBUNTU OS AND ISCSI FEATURES.
CONTENT Introduction... 3 Concepts... 3 Ingredients... 3 Vmware Server 2... 4 Ubuntu Server 10.04... 4 Ubuntu aditional packages... 4 Data Ontap simulator 7.3.4... 4 Netapp licenses... 4 Model... 4 Recipe... 5 Installing licenses... 7 provisioning a lun through iscsi... 7 About me... 10 2 Hernán J. Larrea www.hernanjlarrea.com.ar
INTRODUCTION I have created this document as it has been a great experience to me working with the Data Ontap simulator. It is very real, building the simulator as it is explained on this guide, will let you practice exporting LUN via iscsi, creating shares, working with snapshots, it is almost an all functional virtualized storage. If you want to learn some storage fundamentals or NetApp fundamentals, gain some practice on the Data OnTap CLI or FilerView, this will help you to set up a very real, working, and all functional lab. So, let s cook. CONCEPTS Some stuff you need to get familiar to understand this guide if you have never worked with storage solutions. If you are already familiar with Storage concepts or with NetApp technologies don t bother reading this section. Storage fundamentals, it would be great first you to have some idea of what a storage box is, different vendors, such as EMC, Hitachi and IBM, NetApp is just one more of them. And what is a SAN, and SAN Connectivity, for real storage environments this is the cornerstone, it is a physical layer composed by FC cables, FC switches and directors, even we are not going to virtualize a SAN (since it is not possible), and we are going to use iscsi instead, it would be great you to read some about other Storage Vendors and most of all about SAN and SAN Connectivity, so you can get an idea the way it is in the real world, and get a better understanding on the gaps between the real world and this simulator. Filer, is the way NetApp call s their storage boxes, doesn t care if they are used only NAS (CIFS and NFS) or only for LUN Mapping, or both, they call their boxes, filers. FilerView, is the admin portal for the storage, an alternative for the CLI. Aggregate, Volume, qtree, LUN, Snapshots, this is what you can do with the disks inside your storage, how to arrange them to reach a technical objective. For example, you arrange your physical disks into Aggregates, but the aggregates are composed by raid groups and these ones by physical disks, understand the loop? Of course, an Aggregate can have more than one raid group. Once your aggregate is created, you can create volumes inside of it, and inside these volumes you can create shares, qtrees, luns or snapshots. But as this is not a NetApp administration guide, I m going to stop here talking about this, and just say these are key concepts to understand in order to be able to operate you NetApp simulated Filer, if you want to read more you can do it at NetApp s NOW site, http://now.netapp.com. Finally, I ve mentioned this couple of times at this point, synthetic hardware, this nothing else but the hardware you add to your VM. INGREDIENTS 3 Hernán J. Larrea www.hernanjlarrea.com.ar
What you will need for your recipe and where to get it. VMWARE SERVER 2 Virtualization software, this is my choice, you can change it for any other that supports a Linux operating system, mounting ISO images in synthetic CD hardware of your VM and more than one Ethernet synthetic interface in your VM. VMWare Server 2 it is free, and you can get it from: http://downloads.vmware.com/d/info/datacenter_downloads/vmware_server/2_0 just register and download. UBUNTU SERVER 10.04 This was also my choice, even it is documented by NetApp that is fully supported by their simulator, I guess you can run another Linux flavor you like. It is also free, at the moment this document was written 10.04 was the latest version available. Download from: http://www.ubuntu.com/server/getubuntu/download no need registering, just select 32 bits Server ISO and download it. UBUNTU ADDITIONAL PACKAGES Some packages you will need, as I choose Ubuntu as the base system I used apt in order to get the packages, but if you choose another Linux flavor you might have to download the sources or precompiled binaries and make them on you system. open-iscsi (necessary for mounting iscsi exported LUNs from your virtualized storage) lsscsi (just to check when an iscsi LUN has been mapped to the system) expect (necessary only for running the simulator in background) vim (just because I like it more than vim-tiny) For Ubuntu choosers, all of them available from the Lucid repository. DATA ONTAP SIMULATOR 7.3.4 The master piece, the simulator itself, you can get it from http://now.netapp.com (for this you must be a NetApp s member, partner or customer, as the company I work for is, I ve been able to download it, anyway there is a registration form available on the site, I haven t tried if registering on through that form makes also the download available). You will get an ISO image file from where you will get all you need for your simulator. At the time this document was written, Data Ontap 8 version had already been released but as it is for 64 bits I preferred to keep working on 7.3.4. Also this new version seems is does not need to run on top a VM, of course the ISO for this version is much bigger than for 7.3.4. NETAPP LICENSES Data Ontap simulator includes all the licenses or the products and features available in your filer, such as Flex Clone, Snapmirror, Syncmirror, iscsi, CIFs, NFS, and so on. You can find them inside the ISO image, under./docs/license_keys.htm MODEL 4 Hernán J. Larrea www.hernanjlarrea.com.ar
First of all, let s take a quick view of what we are going to build and its layers (on the following diagram I m going to include only the relevant layers). lun snaps vol0 Volume (vol1) aggr0 Aggregate (aggr1) Data Ontap ns0 eth0 Ubuntu VM iscsi eth1 Data Ontap root Aggregate and Volume Starting from bottom to top, our VM, a common VM configured to run a Linux OS with 2 virtual NICs. We will need 2 because by definition, the Data Ontap simulator, binds its network interface/s to physical interfaces (actually this is a synthetic NIC but the simulator doesn t know that). And once the ns0 interface from the storage is attached to the physical one, the server where the simulator is running it won t be able to access the storage through that interface, and as we are going to map luns though iscsi a second interface will help us to do so. The VM NICs can be bridged, NAT, or host only connections to your hosting machine, that s your call, and also based on what your Virtualization software allows you. As we are going to download some packages with apt, I recommend you to set at least one of them as bridged or NAT. Moving forward, the OS, Ubuntu is this case, we will install Ubuntu Server with the Virtual Machine mode, so a minimal system is installed, no graphical interface, no unneeded packages, no wasted space. Finally the Data Ontap simulator and the open-iscsi package for LUN Mappings. We are talking about this much in depth in the following sections. RECIPE Here you will find the steps for the implementation of the VM and the Simulator, on some parts I ll show a high level description on what should be done, such as in the OS installation process, the guide is meant to let you setup your lab to learn some storage and NetApp not Linux, internet is plenty of documents to answer your questions and I ll be pleased to answer any consult. Also the storage once installed will be simple configured, no clustering, no virtual interfaces, you can play with that if you want, but I m not going to show that here. 1. Create your VM with 2 NICs, 1 configured as NAT or bridged at least. I recommend you to use between 256 and 512 MB for RAM and 8 GB for disks will be ok. 2. Install Ubuntu server as minimal virtual machine mode, during the first screen, ater selecting language, press F4 and select the installation mode. 3. Follow the installation steps for the OS. 4. Take note of which network interface will you select as primary when the installation prompts. 5 Hernán J. Larrea www.hernanjlarrea.com.ar
5. When prompted about server functions to install, select openssh and Samba, for remote administration and filesharing with your desktop, these functions are not required pieces for the puzzle but might make your life easy working with the simulator. 6. Once Ubuntu is installed I recommend you to take a snapshot of the VM, just in case something goes wrong you do not have to run the installation all over again. 7. If installed openssh configure it to accept connections over any of the network interfaces. 8. Edit /etc/network/interfaces to include your second interface. And if you want fix IPs for both interfaces. 9. Restart networking daemon. 10. Install the following packages: a. vim ; Run: apt-get install vim b. open-iscsi ; Run: apt-get install open-iscsi c. expect ; Run: apt-get install expect d. lsscsi ; Run: apt-get install lsscsi 11. Configure iscsi daemon to start automatically. Edit /etc/iscsi/iscsid.conf under Startup Settings, change the startup mode to automatic. 12. Mount the Data Ontap simulator CD and run the installation script./setup.sh following steps (13 to 26 ) show the options given during the setup.sh script execution. Text between brackets shows the default option, if you press return without entering any character this will be the configured option. For these steps, comments will be in bold to make difference between installation prompts and comments. 13. Where to install to? [/sim]: 14. Would you like to install as a cluster? [no]: I ll write another document regard clustering options. 15. Would you like full HTML/PDF FilerView documentation to be installed [yes]: n 16. Continue with installation? [no]: y 17. Your simulator has 2 disk(s). How many more would you like to add? [0]: 26 18. What disk size would you like to use? [a]: a Choose whether you feel comfortable with. 19. Disk adapter to put disks on? [0]: 20. Use DHCP on first boot? [yes]: 21. Ask for floppy boot? [no]: 22. Which network interface should the simulator use? [default]: eth1 Make sure to check an interface different from the one you selected as default during the installation. Also to get this working properly so you can access, the internet, the local network and the simulator from the VM ensure the eth0 is first in the routing table (Run command: route) if not, just delete the routes for eth0 and add them again. 23. How much memory would you like the simulator to use? [512]: 256 Set the amount of memory you feel comfortable with inside the parameters given. 24. Create a new log for each session? [no]: 25. Overwrite the single log each time? [yes]: 26. Run /sim/runsim.sh to start the simulator. The first time you run the simulator it will ask you to set settings relevant to your virtual storage functionality, anytime you want to run this configuration again, you can run /sim/steup.sh. 27. Run /sim/runsim.sh to start the simulator and the first configuration. For next steps (28 to 48) I m going to show the prompts by the setup script, any comments will be made in bold to avoid confusion between script s prompts and comments. 28. Please enter the new hostname []: ontap01 Just choose a name that likes you. 29. Do you want to enable IPv6? [n]: 6 Hernán J. Larrea www.hernanjlarrea.com.ar
30. Do you want to configure virtual network interfaces? [n]: 31. Please enter the IP address for Network Interface ns0 [192.168.0.106]: 32. Please enter the netmask for Network Interface ns0 [255.255.255.0]: 33. Please enter media type for ns0 {100tx-fd, auto} [auto]: 34. Please enter the IP address for Network Interface ns1 []: 35. Would you like to continue setup through the web interface? [n]: 36. Please enter the name or IP address of the IPv4 default gateway [192.168.0.1]: 37. Please enter the name or IP address of the administration host: 192.168.0.101 38. Please enter timezone [GMT]: 39. Where is the filer located? []: Argentina 40. What language will be used for multi-protocol files (Type? for list)?:en_us 41. Do you want to run DNS resolver? [n]: 42. Do you want to run NIS client? [n]: 43. Do you want to configure the Shelf Alternate Control Path Management interface for SAS shelves [n]: New password: Retype new password: 44. Do you want to make the system visible via WINS? [n]: 45. A filer can be configured for multiprotocol access (...) Selection (1-2)? [1]: 1 Enter the password for the root user []: Retype the password: 46. The default name for this CIFS server is 'ONTAP01'. Would you like to change this name? [n]: 47. Data ONTAP CIFS services support four styles of user authentication. Selection (1-4)? [1]: 4 48. What is the name of the Workgroup? [WORKGROUP]: 49. At this point you should be being requested to enter the root password. 50. Once loged in as root you will be taken to the Data Ontap prompt your storage name > in my case Ontap01>. Now you can start playing with your storage! INSTALLING LICENSES For most things we want to do here, we need to install some licenses, from your linux machine mount the CD image and browse to./doc and open the file named license_keys.htm. To install the licenses it is really simple, we can do it through CLI or via FilerView. For our next objective we will need to install iscsi license, so go to the file, grab the code and run the following command in the Data Ontap CLI: license add <code> Repeat the procedure for every product or feature you want to enable. To get a summary on the licenses installed just run the license command. PROVISIONING A LUN THROUGH ISCSI At this point your simulator should be up and running and you should have all disks as spares, to accomplish this objective what we need to now is: Create a new Aggregate. Create a volume inside the Aggregate. Create a LUN inside the volume. Create an Initiator group and add our iscsi address. 7 Hernán J. Larrea www.hernanjlarrea.com.ar
Map the lun and the initiator group. Log in from the OS to the storage over iscsi and discover the LUN. Make a FS inside the new disk and mount it. Keep playing with your storage! I m going to describe how to do this through the CLI, but you might want to do it through the FilerView all options are available from any of these interfaces. So, as we already cooked, let s decorate the plate Creating a new aggregate First of all, as I mentioned in the concepts section, we know disks are arranged into raid groups, and aggregates are composed by raid groups. First of all, lets check the status of our disks. Ontap01>sysconfig r You will get an output with the details on the physical disks and how are they arranged in through the different aggregates. (If no aggregate was created only aggr0, the root aggregate will be shown an all the other disks will be shown as spares). Ontap01>aggr create aggr_tst -r 8 -t raid4 16@16m This command will create an aggregate called aggr_tst with a raid group size of 8 with only 1 disk of parity per raid group (raid 4) and will include 16 disks of 16 MBs each. (So it will create 2 raid groups with 8 disks each, but only 7 will be usable since each raid group has 1 disk or parity). Creating a volume inside the aggregate Now the aggregate is created, check the space on it: ontap01> df -h-a aggr_tst Aggregate total used avail capacity aggr_tst 191MB 80KB 191MB 0% aggr_tst/.snapshot 10MB 0MB 10MB 0% There are 10MB (5%) reserved for aggregate snapshots as we won t take any snapshot from the aggregate let s claim that space: ontap01> snap reserve -A aggr_tst 0 ontap01> df -h-a aggr_tst Aggregate total used avail capacity aggr_tst 201MB 80KB 201MB 0% aggr_tst/.snapshot 0TB 0TB 0TB ---% Now we ve got all the space for us, let s create a 100MB volume inside this aggregate: ontap01> vol create vol_tst aggr_tst 100m This command will create a 100MB volume named vol_tst inside the aggregate aggr_tst. So, the volume is created, let s check the space on it: ontap01> df -h vol_tst 8 Hernán J. Larrea www.hernanjlarrea.com.ar
Filesystem total used avail capacity Mounted on /vol/vol_tst/ 80MB 88KB 79MB 0% /vol/vol_tst/ /vol/vol_tst/.snapshot 20MB 0MB 20MB 0% /vol/vol_tst/.snapshot Again we see space reserved for snapshots, we might want to take snapshots of this volume, so let s keep it, and create a LUN inside this volume. Creating the LUN Just run the following command: ontap01> lun create -s 50m -t linux /vol/vol_tst/lun_tst ontap01> lun show /vol/vol_tst/lun_tst 50m (52428800) (r/w, online) And the LUN is created. The lun create command includes the size of the lun (of course it can t be bigger than the volume where it is located), the type o OS which will access that LUN and finally the path where you want to store it inside your filer. Creating the iscsi initiator group First of all, on the storage side, check the iscsi service is running: ontap01> iscsi status iscsi service is running In order to create the initiator group, you first must know the initiator s iscsi node name, go to the Linux machine and read the content of /etc/iscsi/initiatorname.iscsi file: InitiatorName=iqn.1993-08.org.debian:01:58cad98ae9c6 Then, create the initiator group with the igroup command: ontap01> igroup create -i -t linux Initiators_tst ontap01> igroup show Initiators_tst (iscsi) (ostype: linux): About the create parameter or igoup command, you must specify it is an iscsi initiator goup type (-i option) and the the host type and the group name. As you can see, it has been created but it s empty, let s add the node name of the server. ontap01> igroup add Initiators_tst iqn.1993-08.org.debian:01:58cad98ae9c6 ontap01> igroup show Initiators_tst (iscsi) (ostype: linux): iqn.1993-08.org.debian:01:58cad98ae9c6 (not logged in) Now, we need to map the LUN to the initiator group. Mapping the LUN to the initiator group Run the following command: ontap01> lun map /vol/vol_tst/lun_tst Initiators_tst 1 and the mapping is done. You indicate which LUN to which group and which id will be set to the LUN being mapped. 9 Hernán J. Larrea www.hernanjlarrea.com.ar
Discovering the LUN on the server s side At this point our storage is presenting the LUN via iscsi to the server, now the server must run the necessary steps in order to discover the LUN. simulator:/$ lsscsi [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0 [2:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda As you see, we only have the cdrom device and the disk where the OS is installed. Let s discover the iscsi target: simulator:/# iscsiadm -m discovery -t st -p 192.168.233.131 192.168.233.131:3260,1000 iqn.1992-08.com.netapp:sn.99923498 Now check or SCSI devices, just to keep a pre discovery view simulator:/# lsscsi [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0 [2:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda Now restart the iscsi service: simulator:/# /etc/init.d/open-iscsi restart * Disconnecting iscsi targets [ OK ] * Stopping iscsi initiator service [ OK ] * Starting iscsi initiator service iscsid [ OK ] ln: target `/lib/init/rw/sendsigs.omit.d/' is not a directory: No such file or directory * Setting up iscsi targets [ OK ] Finally, list SCSI devices again: root@simulator:/# lsscsi [1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0 [2:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda [3:0:0:1] disk NETAPP LUN 7340 /dev/sdb Creating an FS inside the LUN Run fdisk and create a new partition inside the LUN and then format the partition. Run: fdisk /dev/sdb, create the partition and write the partition table to the disk. Then run mkfs.ext4 (this is just another one of my choices), Run: mkfs.ext4 /dev/sdb1 Finally mount the partition, Run: mount /dev/sdb1 /mnt. ABOUT ME I m Hernán J. Larrea, an IT guy, I ve been working in IT for almost 7 years. I decided to address my career in Wintel technologies, specially Active Directory, Backup and Recovery environments, Storage solutions and a little programming in.net. If you want to know more about me, I invite you to visit my personal website www.hernanjlarrea.com.ar and if you want to email me, you can do it at hjlarrea@hotmail.com Hernán J. Larrea 10 Hernán J. Larrea www.hernanjlarrea.com.ar