Build Nested Openstack Using Ubuntu as BaseOS on BareMetal
Overview The following slides outline how to setup your BareMetal HOST (a server) in order to run a Openstack installation using FUEL based Deployer. The overall tasks can be broken down into the following sections: Host OS installation and preparation Virtual Machine creation for Openstack Nodes (Compute/Controller and FUEL) Addition of vrouter (in order to isolate a useable PUBLIC network) Setup of VNC to Nested Host (in order to use UI s and convienence) Booting/Configuration of FUEL VM (used to install Openstack to Compute/Controller) Creation of an Environment and deployment of Openstack (job done)
Conceptual Diagram Below you can see what the internal Layout of a Nested installation can look like. In this example, we will use 1 bridge for each of the Openstack Networks (5 in total ADMIN/PXE, PUBLIC, PRIVATE,STORAGE and MANAGEMENT) LMCDASM ADD PICTURE HERE
Host OS and preparation You should perform the following tasks in this section. 1 st Install Ubuntu to your Host and ensure it has connectivity to the internet Note: For the example used here we use Ubuntu 14.04.03 LTS however,the specific base OS is up to the user (note: Centos/RedHat command will be different to reflect systemd usage and are not reflected herein) Note: Depending on the size of your host (Disk/CPU/RAM) some of the values you see in the following screen shots may need to be modified to accommodate your system. Specifications used in this example (this is a small setup for example you should not use this in production at all) HOST HAS: 12 Cpus (Cores) 16GB RAM 250GB disk 1 Physical NIC connected to the internet ISO for booting HOST: ubuntu-14.04.3-server-amd64.iso Note: in this example the HOST is assigned an IP via DHCP this might not be the case in your setup, but it is assumed you know how to setup an IP address if not, then this procedure is going to be a real challenge for you.
Host Install When you boot your HOST with Ubuntu you should see the startup page. In general, we select all the defaults, but below you find the answers to each of the questions. 1 st - Lang English (or whatever you like) 2 nd Install Ubuntu Server (hit enter) 3 rd English 4 th US 5 th No (don t detect Keyboard unless you need it) 6 th English 7 th English 8 th Pick a hostname (ex NestedOstack) 9 th Full Name (eg user) 10 th Username (eg. user) 11 th Password (eg system) 12 th Re-enter Password (eg system) note: Answer yes to use password if not complex enough 13 th Answer no to encrypt home dir 14 th Answer yes if Timezone is correct 15 th Answer Guided use entire disk and setup LVM 16 th Select the Disk to install to. (Hit enter if there is only one) 17 th Answer save Changes and configured LVM 18 th Hit enter to use the total amount of disk (change if appropriate) 19 th Move to YES and hit enter to write all final changes to the disk 20 th Enter Proxy info (if needed). 21 st No to Automatic Updates 22 nd For Software to install Select SSH only (note: We will add the Virtual Machine packages after the installation along with other tools) 22 nd Yes to Grub loaded and reboot the node. Ubuntu should install now wait until it is completed and you have a login prompt and then continue with the HOST-POST-INSTALL steps (next page)
Host Post-Install Login to the node following the installation and perform the following steps. 1 st sudo apt-get update && sudo apt-get upgrade 2 nd sudo apt-get install kvm virt-manager bridge-utils vlan tightvncserver xfce4 xfce4-goodies firefox (note: you can use whatever desktop you like if you don t like xfce4) 3rd Reboot the node (probably don t have to, but a good idea at this point since initramfs was updated in the previous step) 4 th run kvm-ok and ensure you see a positive output if not STOP and get HELP)
Create Bridges As outlined, we will create fives bridges on this HOST, they will represent switches that will hold each of the OPENSTACK networks required (ADMIN/PXE, MGMT, STOR, PUBLIC and PRIVATE). As well, we will create an EXT_BR that will be encapsulate our real network card (so we will be able to route in/out of the Openstack Later on). In this example, we will use 5 FLAT networks no tags - although they can be used just as easily. Note: sudo is implied below (or do sudo I and execute as root) Commands (to be done on console or you will lose connection): Ifconfig virbr0 down brctl delbr virbr0 brctl addbr ext_br brctl addbr admin brctl addbr public brctl addbr private brctl addbr storage brctl addbr mgmt Bring up the bridges Command: ifconfig mgmt 192.168.10.253/24 up ifconfig public 172.16.0.253/24 up ifconfig private up <= Left without IP intentionally ifconfig storage192.168.20.253/24 up ifconfig admin 10.20.0.253/24 upup Encapsulate external NIC to EXT_BR and assign IP to the BRIDGE. 1 st get your current IP address (assuming you have eth0 as your external NIC) Command: ifconfig a eth0 2 nd put your eth0 NIC to a null address (leave it up though) and add it to ext_br and set the IP on the bridge Command: ifconfig eth0 0.0.0.0 brctl addif ext_br eth0 ifconfig ext_br <ip from eth0 original) eg. 192.168.1.115 3 rd Test that you can ping your GW still (eg, my GW is 192.168.1.1) IF this doesn t work STOP and get HELP, cause nothing else will work going forward.
Make Network Setting Permanent Login to your node (you can now use SSH through the ext_br you setup in the previous step) and modify /etc/network/interfaces as shown below user@nestedostack:/etc/network$ cat interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet manual auto ext_br iface ext_br inet static address 192.168.1.115 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 8.8.8.8 bridge_ports eth0 Reboot the node and make sure you can get back in without issue.
Setup VNC server and xfce4 For the VM creation part, I like to use virt-manager (a GUI based tool), you can use the virsh CLI if you like, but the GUI allows to see the CONSOLE of the VMs and edit SPICE and other params on the fly. The following section outlines how to configure tightvncserver and xfce4 for use. Step 1 Login to your HOST and start VNCSERVER and set password (this is the password you will use to connect from your VNC client) DO NOT RUN AS ROOT. Command: vncserver: You will be prompted for a password (you can decide if you want a read-only one as well) You should see the server started on :1
Setup VNC server and xfce4 /2 Now kill the running vncserver Command: vncserver kill :1 Modify your users /home/<user>/.vnc/xstartup file to look like this (remove everything else) #!/bin/sh unset SESSION_MANAGER unset DBUS_SESSION_BUS_ADDRESS startxfce4 & [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources xsetroot -solid grey vncconfig -iconic & Restart your VNC server (note: :1 is port 5901 for connecting with your client) Command: vncserver Sample Output: user@nestedostack:~$ vncserver New 'X' desktop is NestedOstack:1 Starting applications specified in /home/user/.vnc/xstartup Log file is /home/user/.vnc/nestedostack:1.log user@nestedostack:~$
Connect with VNC client I use UltraVNC as a client, but you can use whatever you like. (you can pull X display across if you have good bandwidth) Open your VNC Client and connect to your Host, screen shot examples below:
Connect with VNC Client /2 Observe a desktop on your HOST is displayed. Select default Config
Create Your VMs In this example, we will create a total of 4 VMs. 1 VM for a vrouter this is used to isolate our Openstack networks and provide routing and access for FUEL to the Openstack repos 1VM for a FUEL server this is the node that will boot and install the Controller and Compute 1VM for Controller 1VM for Compute Note: You can create as many VM s as your HARDWARE allows, just repeat the Section for COMPUTE/CONTROLER (note: Max 3 Controllers for HA, Min 1)
Create VM - vrouter /1 With our VNC open, open a terminal and start virt-manager, see screen shot below:
Create VM vrouter /2 For the vrouter, I use Vyatta (you can get the image at http://dasmlab.org/vyatta-livecd_vse6.7r6_amd64.iso), or you can use VyOs, or Quagga, or whatever you like. Push the image to your host (SFTP/HTTP/etc) so it is accessible there. In the virt-manager GUI create your VM Details: 1 CPU, 2GB RAM, 5GB HD, 3 NICs (on ext_br, admin, public) The following SCREEN SHOT outline each STEP
Create VM vrouter /3 Click Create New Virtual Machine
Create VM vrouter /4 Give your VM a name as shown and select Local media install :
Create VM vrouter /5 Point to the ISO location and set the OS_TYPE and Version (pick any for version doesn t matter) click Forward. Set RAM/CPU as shown and click Forward. Set disk to 5GB and click Forward
Create VM vrouter /6 On the next page, CHECK the customized box, and ensure that the Advanced Options is outlined as shown below (must be on ext_br) and click Finish.
Create VM vrouter /7 On the Virtual Machine Details page (shown) below you will do the following tasks (use the ADD HARDWARE BUTTON for NICs): 1 st Change Display VNC from SPICE to VNC. 2 nd Add two more NICs one on bridge admin and one on bridge public
Create VM vrouter /8 Picture showing addition of NIC to admin bridge (note: you must specify the name
Create VM vrouter /9 Picture showing adding NIC to public bridge
Create VM vrouter /10 Verify your screen matches below and click Being Installation
Create VM vrouter /11 Console open on your VM, and will boot. Once you are at the login prompt, execute the following commands to setup your vrouter 1 st Login (vyatta/vyatta) 2 nd Command: install image Responses: Yes, Auto, vda, Yes, <enter>, <enter>, <enter>, vyatta, <enter> 3rd Click on the! at the top of the Console, details page will be displayed Select Boot Options and change order to boot from Disk. Select Apply and then reboot the VM (see screen on next page for example). IF YOU DON T you will only boot from ISO again and nothing will be permanent in your vrouter.
Create VM vrouter /12 Screen shot of Changing Boot Order
Create VM vrouter /13 Now that your Router is rebooted off its Hard disk, perform the following configurations. Note the difference between config and command mode in vyatta Command (to be executed in order once you login) 1 st config and press Enter (you should see edit above prompt) 2 nd set interfaces ethernet eth0 192.168.1.116/24 <= where this is an IP on the external subnet that your hosts physical NIC is on 3 rd set interfaces ethernet eth1 10.20.0.1/24 4 th set interface ethernet eth2 172.16.0.1/24 5 th set nat source rule 1 destination address 0.0.0.0/0 6 th set nat source rule 1 outbound-interface eth0 7 th set nat source rule 1 protocol all 8 th set nat source rule 1 source address 10.20.0.0/24 9 th set nat source rule 1 translation address masquerade 9.1 set service dns forwarding listen-on eth1 9.2 set service dns forwarding name-server 8.8.8.8 10 th commit 11 th save Move onto the next page
Create VM vrouter /14 Continue configuring your vrouter with the following commands: 1 st set nat source rule 2 destination address 0.0.0.0/0 2 nd set nat souce rule 2 outbound-interface eth0 3 rd set nat source rule 2 protocol all 4 th set nat source rule 2 source address 172.16.0.0/24 5 th set nat source rule 2 translation address masquerade 6 th set protocols static route 0.0.0.0/0 next-hop 192.168.1.1 <= where the 192 is the IP of my GW yours might be different 7 th set system name-server 8.8.8.8 <= where the IP is of your DNS server 8 th set system ntp server 0.ca.pool.ntp.org <= where NTP is a valid address you MUST have a good NTP source LCL is a bad idea in VMs. 9 th commit 10 th save YOUR VROUTER IS CONFIGURED AND READY TO GO NO MORE WORK THERE
Create VM - vfuel The following steps will create a FUEL VM that is used to create/deploy and manage your Openstack Installation. For this we will use the Brahmaputra (B-release) of FUEL from OPNFV as our ISO to creation you can fetch this at: http://artifacts.opnfv.org/fuel/brahmaputra/opnfvbrahmaputra.1.0.iso Download that ISO on your host, it will be need in the following steps
Create VM vfuel /2 Login to your HOST, open virt-manager and click Create New Virtual Machine. Your first screen should like this then click Forward
Create VM vfuel /3 On the following page, point to the FUEL ISO you downloaded and set the OS_TYPE and OS_Version as shown below, click Forward.
Create VM vfuel /4 Send the RAM/CPU as shown. Set the DISK size as shown.
Create VM vfuel /5 Set the final configuration screen as shown below ensure you put the NIC on the ADMIN bridge (specify bridge name) as shown below and click Finish.
Create VM vfuel /6 The FUEL VM will boot the splash screen show below Select FUEL INSTALL (Static IP) and press Enter. The node will install will take some time.
Create VM vfuel /6 Once your VM reboots, you will be presented with this screen press enter to answer the question about password. We will have to modify a couple values in this menu before proceeding.
Create VM vfuel /7 Navigate (arrow keys) to DNS & Hostname section and ensure that the DNS entry is valid for your setup. You can modify the hostname and dns suffix to suit your needs (not required for vpod). Once edited, move to Check and press Enter.
Create VM vfuel /8 Navigate to Time Sync section and configure the appropriate NTP server NOTE: for vpod, a valid NTP server is critical Click check
Create VM vfuel /9 Navigate to Quit Setup and Select Save and Quit. The Fuel VM will test that your configuration is correct and if all is well, finish the installation.
Create VM vfuel /10 When you FUEL install is complete the Console will show the following:
Create VM vfuel /11 Point your LOCAL browser (in your VNC session, or X exported firefox) to http://10.20.0.2 and login (admin/admin). You should see the following.
Create VM vfuel /12 After your first login, uncheck Send Stats and click Connect Later, you should have a page like this. Continue to the next steps
Create VM vcontroller The following steps outline how to create the vcontroller (your Openstack Controller node). This node will be jumpstarted by the FUEL node, so we only need to create the VM, the NICs and a empty hard disk no image is supplied. As before, login to your HOST and open virt-manager and your first screen should look like this (note: We use PXE BOOT here):
Create VM vcontroller /2 One the following screen, select the OS type and Version, RAM and CPU
Create VM vcontroller /3 Ensure that you have the following items checked on the summary screen (specify admin for the first bridge) we will add four more NICs in the Details Screen
Create VM vcontroller /4 Add four more Network Cards to the Controller, one for each network (bridge) in the setup (mgmt., storage, public, private).
Create VM vcontroller /5 Screen shots of adding last two NICs
Create VM vcontroller /6 Below is an example of the layout of the vcontroller, when satisfied, click Begin Installation. The VM will boot and the vfuel will provide PXE boot. Nothing else needs to be done on Controller at this point
Create VM vcompute /1 The following steps outline how to create the VM for the COMPUTE. These steps are identical to the vcontroller, except that you need to edit the cpu_model in the vcompute configuration to ensure KVM flags are passed from HOST to GUEST (so that the COMPUTE can create VM s itself). Notes: In a virtual environment the more storage,ram and CPU you give to the vcompute translates into more resources for Instance (VM) creation under Openstack.
Create VM vcompute /2 Create the VM as before ensure you select PXE BOOT on the naming page.
Create VM vcompute /3 Configure you Storage appropriately and make sure you select admin as the first bridge for this VM and check to configure options. Note: Ignore Network does not support PXE it Lies! :P)
Create VM vcompute /4 As we did with the Controller, add 4 more NIC s to this VM (mgmt, storage, private,public) respectively so that you screen resembles this
Create VM vcompute /5 There is a bug in the latest kvm/virt-manager and you need to configure CPU information manually. Open the Processor tab of the vcompute and navigate to Configuration and click Copy Host CPU Properties. Example shown below, click Apply and then Begin Installation
Create VM vcompute /6 After you click Begin Installation the VM will boot, PXE boot from the FUEL node and you will end up at a bootstrap login prompt (shown below). You do not need to do anything else on the vcompute at this point.
Create Openstack Environment The following steps are executed within the FUEL UI (Web Page). Login to the FUEL Web page at (http://10.20.0.2) (admin/admin) Observe in the Top right Corner that you have 2 (or more) nodes that are available to be used. Example below
Create Openstack Environment /2 In the Main Page (click Environments at the top if you get lost). Click New Openstack Enviroment Provide : A Name for your Environment e.g TEST ENV and Click Next Select Hypervisor Type (QEMU/KVM or vcenter) and Click Next Select Networking Type (VLAN or VXLAN) and Click Next Select Storage Backends (LVM or CEPH note: CEPH needs min 3 nodes) and click Next Select Additional Openstack Options desired and Click Next Click Finish and your Environment will be initialized and you will have a screen presented (see next page)
Create Openstack Environment /3 ENV screen following FINISH from last step.
Create Openstack Env /4 Click on Node Tab from the Environment Screen Click Add Nodes Select Role Controller (check box) and then select your Controller node from the list at the bottom (use the MAC if you don t know which is which) Click Apply Changes Click Add Nodes Select Role Compute and Storage-Cinder (check box) and then select your COMPUTE node from the list below (check box). Click Apply Changes
Create Openstack Env /5 You screen should look similar to the following:
Create Openstack Env /6 On the Nodes Screen Check Select All Then Click on Configure Interfaces Drag and Drop the Networks so that they match the screen shot below and Click Apply
Create Openstack Env /7 Click on the Networks Tab on your Environment Screen (next to Nodes) and perform the following modifications. Under Storage: change CIDR to 192.168.20.0/24 Uncheck VLAN tag (not used) Under Management: Change CIDR to 192.168.10.0/24 Uncheck VLAN Tag (not used) Click Save On the Left Click Neutron Layer 3 and update the DNS with a valid entry (should be the same as you used in the FUEL step) and click Save On the Left click Verify Connectivity and then click the Verify Networks button. This is vital and if this fails, your deployment will correct any errors seen and re-run until you see Verify Network Succeeded see screen shot on next Page
Create Openstack Env /8 Screen shot example of verify networks success
Create Openstack Env /9 Now that Verify has passed, click on the Dashboard TAB (to the right of Nodes) in the FUEL UI and click Deploy. This will take between 30-60 minutes to deploy and when completed, you will be able to access your Openstack via Horizon (URL will be shown on the screen) or CLI (via the FUEL node): Note: By default, SSHD to the COMPUTE/CONTROL is disabled except from the FUEL so you need to SSH to your FUEL (root/r00tme) and then to the openstack nodes. You can run fuel node when on the vfuel and see the admin IP s of your nodes (or use the hostname)
Create Openstack ENV /10 Screenshot of deployment Tab