Quantum Policy Suite Operations Guide

Transcription

1 Quantum Policy Suite 5.5. Operations Guide Version 5.5. September 30, 03 Cisco Systems, Inc. Cisco has more than 00 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at Text Part Number:

2 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB s public domain version of the UNIX operating system. All rights reserved. Copyright 98, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0R) Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. 03 Cisco Systems, Inc. All rights reserved.

3 CONTENTS CHAPTER Preface - Readers - Additional Support - Terms and Definitions - CHAPTER Quantum Policy Server Operations - Starting and Stopping - Restarting the Quantum Policy Server - Shutting Down the Quantum Policy Server - Starting VMware -3 Rebooting a VM -3 Switching Active and Standby -3 Determining the Active Load Balancer -4 Determine the Standby Load Balancer -4 Determine Standby Function and Statefulness -4 Backing Up and Restoring -4 Backing Up and Restoring the Databases -4 Backing Up SVN -4 Adding or Replacing Hardware -5 Publishing Data -5 Version Control Software -5 Setting Debug Levels -6 Synchronizing Times -7 Minor Time Synch Result -8 Major Time Sync Result -8 Quantum Policy Server Deployment Start to Finish -8 VMware vsphere Hypervisor Install and Configuration -9 Install VMware ESXi Configure VMware ESXi OVF Template Deployment -6 Install vsphere Client -6 OVF Template Location -6 Import OVF Templates -6 iii

4 Contents Managing Subscriber Data -0 Subscriber Management with AAA -0 Subscriber Management with USuM -0 Additional Documents -0 CHAPTER 3 Updating the Installation 3- Initial Upgrade 3- Normal Update Procedure 3- CHAPTER 4 Expanding the Deployment 4- Expanding Architecture for Scalability 4- Typical Scenarios When Expansion is Necessary 4- Hardware Approach to Expanding 4- High Availability (HA) Consequences 4- Adding a New Blade 4- Component Approach to Expanding 4-3 Adding Additional Component 4-3 Expanding Architecture for Geographic Redundancy 4-4 Data Synchronization 4-4 Active-Standby Mode 4-6 Active-Active Geographic Clusters 4-6 Session Manager Configuration 4-8 Verifying Session Manager Cluster 4-8 Policy Director Configuration 4-3 CHAPTER 5 Quantum Policy Server Blade Hardware 5- Server Prerequisites 5- CHAPTER 5 5- G LOSSARY APPENDIX A File Examples A- Example esx.conf File A- Example qns.conf File A-3 iv

5 CHAPTER Preface Revised: September 30, 03, Welcome to Cisco s for Quantum Policy Server. This document describes operations, maintenance, and troubleshooting activities for the various VM servers in the Cisco s Quantum Policy Suite. This document assists system administrators and network engineers to operate and monitor the Quantum Policy Server 5.5 and its parts. This preface covers the following topics: Readers Additional Support Terms and Definitions Readers This guide is best used by these readers: Deployment engineers System administrators Network administrators Network engineers Network operators Implementation engineers This document assumes a general understanding of network architecture, configuration, and operations. Instructions for installation and use of QPS 6.0 and related equipment assume that the reader has experience with electronics and electrical appliance installation. Additional Support For further documentation and support: Contact your Cisco, Inc. technical representative. Call the Cisco, Inc. technical support number. -

6 Terms and Definitions Chapter Preface Write to Cisco, Inc. at Refer to your other documents. Terms and Definitions This document uses certain terms and definitions specific to the QPS software application. Please refer to our common Glossary. -

7 CHAPTER Quantum Policy Server Operations Revised: September 30, 03, This chapter covers the following topics: Starting and Stopping Switching Active and Standby Backing Up and Restoring Adding or Replacing Hardware Publishing Data Version Control Software Setting Debug Levels Synchronizing Times Quantum Policy Server Deployment Start to Finish Managing Subscriber Data Additional Documents Starting and Stopping On the advisement of your support staff, you may want restart the Quantum Policy Server because of troubles seen in logs or perhaps performance issues. This section describes several start and stop tasks for the Quantum Policy Server. Restarting the Quantum Policy Server Shutting Down the Quantum Policy Server Starting VMware Rebooting a VM Restarting the Quantum Policy Server This operation stops and restarts the software modules and components. -

8 Starting and Stopping Chapter Quantum Policy Server Operations Step Restart sessionmgr, the session database, with these commands: service sessionmgr-777 stop service sessionmgr-777 start service sessionmgr-778 stop service sessionmgr-778 start service mongodb-779 stop service mongodb-779 start service mongodb-770 stop service mongodb-770 start service mongodb-773 stop service mongodb-773 start Step Restart the Quantum Policy Server with this command: service qns restart Step 3 Restart SVN with this command: service httpd restart Step 4 Restart the Quantum Policy Builder client with these commands: service qns restart Step 5 Restart Load Balancer with these commands: RADIUS: service qns restart SSL: service stunnel restart DHCP: service ldirectord restart HA VIP Failover: service heartbeat restart All Other: service haproxy restart Shutting Down the Quantum Policy Server If you need to shut down the Quantum Policy Server, you can do so from the console or logged in as the root user. To shutdown from the Linux command line, log in as root and use the init command: init 0 -

9 Chapter Quantum Policy Server Operations Switching Active and Standby Starting VMware Step Step Start a VMware vsphere session. Right-click on the VMware ESXi IP address on the left and select Power On. Rebooting a VM Occasionally, you may be asked to bounce a VM server. To stop and restart a VM from the Linux command line, use the init command as shown: init 6 Switching Active and Standby In QPS, the active and standby strategy applies only to the secondary (0) load balancers. Recall that the four load balancers are in the system are these: lb0 lb0 portallb0 portallb0-3

10 Backing Up and Restoring Chapter Quantum Policy Server Operations Determining the Active Load Balancer For the couples lb0 and lb0 and portallb0 and portallb0, the active load balancers have the Virtual IPs (vips) active. Run the command ifconfig to return the addresses assigned to the interfaces eth0:0 (or eth0:) and eth:0 (or eth:). Equivalently, run the command ip addr to return secondary IP addresses (the VIPs) assigned to eth0 and eth. Determine the Standby Load Balancer The passive or standby load balancer is the system which does not have active VIPs. Determine Standby Function and Statefulness The standby load balancer is available via tcp/udp (or icmp). Ping or ssh to the machine. The standby load balancer shows running... as the return to the command service heartbeat status. Backing Up and Restoring As a part of routine operations, you will need to have backups and may need to restore them. This section describes several backup and restore tasks for the databases for Quantum USuM, SessionMGR, and SVN. Backing Up and Restoring the Databases Backing Up SVN Backing Up and Restoring the Databases For more information, see the document Quantum Policy Suite 6.0 Backup and Restore Guide. Backing Up SVN Backup up the SVN repository is performed within SVN itself. Step Step Step 3 Step 4 Download and install Tortoise SVN (or any other SVN client). Connect to Check out the configuration and run directories. The run directory contains the runtime information for the Quantum Policy Server. The configuration directory contains the Quantum Policy Builder client information. The default login for this is broadhop/broadhop. -4

11 Chapter Quantum Policy Server Operations Adding or Replacing Hardware Adding or Replacing Hardware Hardware replacement is usually performed by the hardware vendor with whom your company holds a support contract. Hardware support is not provided by Cisco. The contact persons and scheduling for replacing hardware is made by your company. Before replacing hardware, see if you have a recent backup. If not, try to make a backup now. See Backing Up and Restoring. Unless you have a readily available backup solution, use VMware Data Recovery. This solution, provided by VMware under a separate license, is easily integrated into your QPS environment. New installations to new hardware can be performed a couple of different ways, but the most common is to install via virtual machine templates VMware ESXi. See Quantum Policy Server Deployment Start to Finish. The templates you download from the Cisco repository are partially pre-configured but require further configuration. Your Cisco technical representative can provide you with detailed instructions. Note You can download the VMware software and documentation from the following location: Publishing Data This section describes publishing Quantum Policy Builder data to the Quantum Policy Server. Publishing data occurs in the Quantum Policy Builder client interface, but affects the Quantum Policy Server. The Configuration Guide for Quantum Policy Builder, discusses publishing to the server in detail. Please refer to that document. You will have to log in to the Quantum Policy Builder to publish data. Quantum Policy Builder manages data stored in three areas: The Client Repository stores data captured from the Policy Builder GUI locally. This is a place where trial configurations can be developed and saved without affecting Quantum Policy Builder server data. The Updates Repository stores information on where to find software updates, including their URIs. This is the area that affects the Quantum Policy Server: The Server Repository stores configuration data about policies, system configuration, and subscriber servers after it is checked into a version control software. The server takes its data from this repository after you have used the Publish option. You are concerned with the Server Repository data for this operation. Version Control Software Before setting up your client repository sites, version control software is installed and available. Users do not use version control software directly. Rather, the Quantum Policy Server uses is for publishing of repository data. -5

12 Setting Debug Levels Chapter Quantum Policy Server Operations In the Quantum Policy Server, a Subversion server is located at QPS comes with the Subversion version control software. You can use your own copy of Subversion server, but not another version control software product such as IBM ClearCase. Setting Debug Levels Setting debug levels for QPS occurs in Zabbix, the third-party monitoring software QPS uses. Open a Zabbix session and set them there. You can also edit the logback.xml file to set QPS debugging levels. Step Edit the logback.xml file in the /etc/broadhop directory and the logback.xml in the /etc/broadhop/controlcenter directory. Start by looking at the /etc/broadhop/logback.xml. It will have a section that looks similar to this: <!-- Configure Loggers --> <!-- Hide 'Could not load class...' noise. --> <logger name="org.springframework.osgi.extensions.annotation.servicereferencedependencybeanfa ctorypostprocessor" level="error" /> <logger name="org.springframework" level="warn" /> <logger name="com.broadhop.resource.impl" level="warn" /> <logger name="com.danga" level="warn" /> <logger name="httpclient.wire" level="warn" /> <logger name="org.apache.commons.httpclient" level="warn" /> <logger name="sun.rmi.tranrsport.tcp" level="warn" /> <logger name="org.apache.activemq.transport.inactivitymonitor" level="warn" /> <!-- Configure default Loggers --> <root level="warn"> <appender-ref ref="file" /> <appender-ref ref="socket" /> </root> Note The level is configurable to error, warn, info, or debug in order of least logging to most logging. When debugging an issue or upon initial installation, it is most helpful to set the logging level to debug. -6

13 Chapter Quantum Policy Server Operations Synchronizing Times Step To change logging level, change one of the levels or add additional categories, which may involve help from a Cisco support representative. Look at the /etc/broadhop/controlcenter/logback.xml. It will have a section that looks similar to this: <!-- Configure Remote Logger --> <logger name="remote" level="info" additivity="false"> <appender-ref ref="consolidated-file" /> <appender-ref ref="consolidated-jmx" /> </logger> Again, it may be helpful to set this level to debug for initial installation purposes, but no other changes are necessary for this file. After your system is up and running, it is most useful to turn the system to either error or warn. The levels debug or info usually have logs rollover very quickly. After the log rolls over, the information is lost. For this reason, warn or error generates a substantially smaller amount of logging, and gives you the ability to look for issues over a longer period of time in the system. Synchronizing Times This section explains how to synch time between all of the QPS services so they all have the same clock reading. System times are synchronized from lb0 and lb0. All VMs point their NTP to lb0 and lb0 for time sync. Both load balancers are preconfigured to look out to internet-based NTP pools. If pools are unavailable, you must set this manually, as follows: If lb0 and lb0 HAVE access to a valid NTP server, log in to lb0 and lb0 as root and run: service ntpd stop ntpdate -b <ntpserverip> Replace <ntpserverip> with a valid IP. This sets the system date. hwclock --systohc This will sync the hardware clock (VMWare BIOS) to the system time. date (to check if the date is actually correct) hwclock (to check if the hw clock is actually correct) Edit /etc/ntp.conf and search for lines beginning with server. There should be 3 lines pointing to NTP pools. Replace these lines by the <ntpserverip> above (i.e., if <ntpserverip>= , then the only server line in the.conf file should read: server service ntpd start If lb0 and 0 DO NOT HAVE access to a valid NTP server, log in to lb0 and lb0 as root and run: -7

14 Quantum Policy Server Deployment Start to Finish Chapter Quantum Policy Server Operations service ntpd stop date -s " ::4 Replace the example date with the real one. This sets the system date. hwclock --systohc This syncs the hardware clock (VMWare BIOS) to the system time. date (to check if the date is actually correct) hwclock (to check if the hw clock is actually correct) service ntpd start Minor Time Synch Result The system times on the remaining QPS servers will slowly, over hours or days, be brought into proper time by NTP if the system time is slightly off, slightly meaning maybe 30 minutes or so. Major Time Sync Result If other system times are off by more than 30 minutes, NTP may never converge times. Better practice is to set the system times manually for each individual server. Run the script below from lb0. Time will be in sync as close as the time it takes you to type the root password. lb0 portal0 sessionmgr0 qns0 portal0 sessionmgr0 qns0 pcrfclient0 portallb0 qns03 pcrfclient0 portallb0 qns04 Log in to each server as listed above and run this command: MYDATE=$(date); ssh $server "service ntpd stop;date -s \"${MYDATE}\";service ntpd start;hwclock --systohc";done Quantum Policy Server Deployment Start to Finish An operation that all deployments use, but usually only once or twice, is the deployment process. This operation is provided here in the event you are asked to reinstall the Quantum Policy Server, perhaps to move it to another piece of hardware. Note In a full-environment, this operation needs to be performed on each of the VMs. Use these two procedures to do a basic Quantum Policy Server deployment at your site: VMware vsphere Hypervisor Install and Configuration OVF Template Deployment -8

15 Chapter Quantum Policy Server Operations Quantum Policy Server Deployment Start to Finish VMware vsphere Hypervisor Install and Configuration Install VMware ESXi 4.+ Step Step Step 3 Step 4 Download and request a free license from VMware at: For further clarification, refer to the VMware vsphere Hypervisor Install Guide at: Burn the ISO to a disk. Boot your server off of the ISO disk. Please refer to your server s owners manual for directions on how to change the boot order. Run the installer and accept all the defaults. Configure VMware ESXi 4.+ Step Step Connect a keyboard and monitor to your newly installed ESXi server. Press F to customize the system. Step 3 When prompted for a user name and password: Type root as the user name. -9

16 Quantum Policy Server Deployment Start to Finish Chapter Quantum Policy Server Operations Leave the password blank. Press OK. Step 4 Select Configure Password from the System Customization menu to configure the password to broadhop. Step 5 The Old Password is blank. For the New Password and Confirm Password, type broadhop. -0

17 Chapter Quantum Policy Server Operations Quantum Policy Server Deployment Start to Finish Step 6 Step 7 Configure the network interface to which you will connect your ESXi client. Select Configure Management Network from the System Configuration menu. Step 8 Make sure you select the correct Network Adapter on which your management interface will reside. -

18 Quantum Policy Server Deployment Start to Finish Chapter Quantum Policy Server Operations Step 9 In this example, there are only two network interfaces. Define the first physical network interface as the management interface. Highlight the physical interface you wish to use and select it with the spacebar. Step 0 Optional: If there is a flat network there is no need to configure a VLAN. To configure a VLAN, select VLAN from the Configure Management Network menu. -

19 Chapter Quantum Policy Server Operations Quantum Policy Server Deployment Start to Finish Step For this example the ESXi host's management interface is on vlan 7. Type your own interface ID and press Enter. Step Setup the IP address for the ESXi server. Select IP Configuration from the Configure Management Network. -3

20 Quantum Policy Server Deployment Start to Finish Chapter Quantum Policy Server Operations Step 3 On the IP Configuration screen, make sure Set static IP address and network configuration is selected, then fill out the IP address, subnet mask, and gateway information. Step 4 Select yes or OK to restart the network management agents. Test your management network and make sure you can ping the default gateway. -4

21 Chapter Quantum Policy Server Operations Quantum Policy Server Deployment Start to Finish Step 5 Select Test Management Network from the configuration screen. Step 6 Enter the gateway IP address and press OK. The reply should come back OK. Step 7 If you get a ping reply, then continue on to OVF Template Deployment. -5

22 Quantum Policy Server Deployment Start to Finish Chapter Quantum Policy Server Operations OVF Template Deployment Install vsphere Client This section helps you deploy or install the OVF templates required to run the virtual machines at your site. Install vsphere Client OVF Template Location Import OVF Templates Step Step Step 3 When your ESXi server is connected to your network, open a browser and point to the IP address of your ESXi server: Click on the Download vsphere Client link. Run the installer and accept the defaults. OVF Template Location To obtain the OVF templates, contact your Cisco technical staff. These templates are kept inside the Cisco firewall on Confluence for security and ease of update. Import OVF Templates Step Start your vsphere Client. -6

23 Chapter Quantum Policy Server Operations Quantum Policy Server Deployment Start to Finish Step Step 3 Enter the IP address of your vsphere Hypervisor Server along with the user name and password. When authenticated, select File > Deploy OVF Template. Step 4 Select the location of the OVF Template on your local machine. Step 5 Press Next to verify the details of the OVF Template. -7

24 Quantum Policy Server Deployment Start to Finish Chapter Quantum Policy Server Operations Step 6 Press Next to enter a name for this OVF Template in the Name field. Step 7 Step 8 Define which datastore to deploy on. In most situations, deploy on datastore. Decide if you want to deploy the images in Thin Mode or Thick Mode and choose that radio button. All VMs can be deployed in Thin Mode except for the SessionMGR VMs. -8

25 Chapter Quantum Policy Server Operations Quantum Policy Server Deployment Start to Finish Step 9 Map the physical interfaces to the virtual networks configured in VMware. This example assumes a flat network with a single network interface, although this value can be changed later. Step 0 Confirm all of your options and press Finish. -9

26 Managing Subscriber Data Chapter Quantum Policy Server Operations Managing Subscriber Data Subscriber Management with AAA Subscriber Management with USuM Subscriber Management with AAA Please see the documentation for your AAA server. Subscriber Management with USuM Please use the documentation that comes with Cisco Unified SuM, available through your Cisco technical representative. Additional Documents Other documents are available for your use as a Quantum Policy Server administrator. In addition to this guide, you may want to have these documents available: Alarming and SNMP Guide QNS Troubleshooting Guide Blueprint Solution documents specific to your enterprise Monitoring Guide -0

27 CHAPTER 3 Updating the Installation Revised: September 30, 03, This chapter describes how to perform software updates for QPS 6.0. Installation steps must be performed before any updates. This chapter covers the following topics: Initial Upgrade Normal Update Procedure Initial Upgrade Use these steps if you have never performed an install. These steps are for an initial installation, that is, from versions of 5.3 pre-feb 0 th, 0. Note Release 5./5. upgrade is currently unsupported. If you have a version obtained before this, use the previous install guide, version.0. If you do not currently have the updated /opt/broadhop/installer directory (check for /opt/broadhop/installer/.git), you can use these installation instructions. In summary: get update tar.gz and cisco-sp-wifi.tar.gz into /var/tmp Unzip and use the stage_updates.sh script inside to bootstrap the install (after this update you can use: /opt/broadhop/installer/stage_updates.sh without unzipping) cd /var/tmp Find the latest yum repository (System RPM updates) wget tar.gz Grab the latest SP Wifi (QPS/Portal/Installer) Software wget 3-

28 Normal Update Procedure Chapter 3 Updating the Installation Note Please contact your local Cisco Technical representative in case you are not able to download the software. Unzip tar zxvf cisco-sp-wifi.tar.gz tar zxvf cisco-sp-wifi/installer-r*.tar.gz Install software installer/update_rpm_repo.sh broadhop_3p_repository-latest.tar.gz Stage updates (say yes to update_vms and update_qps) installer/stage_update.sh cisco-sp-wifi.tar.gz Normal Update Procedure Use these steps if you have performed an initial installation as above. Stash existing changes - needed until scripts are updated Grab the latest SP Wifi cd /var/tmp Note Before doing the wget below, make sure that a cisco-sp-wifi.tar.gz files does not exist or the wget will create a cisco-sp-wifi.tar.gz. wget Note Please contact your local Cisco Technical representative in case you are not able to download the software. Tell QPS to update (say 'y' to all steps) /opt/broadhop/installer/stage_update.sh /var/tmp/cisco-sp-wifi.tar.gz 3-

29 CHAPTER 4 Expanding the Deployment Revised: September 30, 03, This chapter covers the following topics: Expanding Architecture for Scalability Expanding Architecture for Geographic Redundancy Note The host addresses used in the examples may be different for your deployment. Expanding Architecture for Scalability For future installations and network upgrades, this section proposes what hardware and components you should consider as you grow your network. Typical Scenarios When Expansion is Necessary Hardware Approach to Expanding Component Approach to Expanding The Quantum Policy Server solution is a robust and scalable software-based solution that can be expanded by adding additional hardware and software components. The following sections explain typical scenarios of when to expand the hardware and software to effect such growth. Typical Scenarios When Expansion is Necessary Your network may grow for the following reasons: The subscriber base has grown or will grow beyond the initial installation specifications. In this case, the number of active or non-active subscribers becomes larger than the initial deployment. This can cause one or more components to reach capacity. New components must be added to accommodate the growth. The services or subscriber scenarios have changed, or new services have been introduced, and the transactions per second on a component no longer meet requirements. When a new service or scenario occurs, often there is a change in the overall Transactions Per Second (TPS), or in the TPS on a specific component. When this occurs, new components are necessary to handle the new load. 4-

30 Expanding Architecture for Scalability Chapter 4 Expanding the Deployment The operator notices that there are factors outside of the initial design that are causing either the overall system or a specific component to have a high resource load. This may cause one or multiple components to reach its capacity for TPS. When this occurs, new components are necessary to handle the new factors. Hardware Approach to Expanding Adding a new component may require adding additional hardware. However, the addition of more hardware depends on the physical resources already available, plus what is needed for the new component. High Availability (HA) Consequences When adding more hardware, the design must take into consideration the high availability needs (HA) of the system. The HA design for a single-site system is N+ at the hardware and application level. As a result, adding a new blade incrementally increases the HA capacity of the system. For example, in a basic installation there are Quantum Policy Server blades handling the traffic. The solution is designed so that if one of the blades fails, the other blade can handle the entire capacity of the system. When adding a third blade for capacity expansion, there are now blades to handle the system load if one of the blades fails. This allows for a more linear scaling approach because each additional blade can be accountable for being able to use its full capacity. Note When adding new blades to a cluster, the blades in the cluster must be co-located to achieve the proper throughput between other components. Adding a New Blade To add new blade hardware, Step Log in to the Control Center 0 component on the Control Center Blade. Step Through SSH or through the vsphere Client console, go to the directory /etc/broadhop/scripts/installation Step 3 Step 4 and make sure the esx.conf file has the extra lines needed for the components. See Appendix A, File Examples for an example of the esx.conf file. When the configuration file changes are complete, run the install_esx.sh script to install the new blade and components. The script recognizes the additional components and installs them into the environment. After the script finishes, go to the new blade(s) and log in using the vsphere Client, and start each component. 4-

31 Chapter 4 Expanding the Deployment Expanding Architecture for Scalability Component Approach to Expanding The most common components to be expanded are on the Quantum Policy Servers. As your system begins to scale up, you will need to add more QPS nodes and more SessionMgrs. Expansion for other components can follow the same pattern as described here. The next sections discuss the configurations needed for those specific components to be active in the system. Adding Additional Component All new components must be added to the /etc/hosts file on Control Center 0 and synchronized to all of the other components using /opt/broadhop/qns/bin/synconfig.sh. Note For specific components, additional configuration is necessary. Additional portal components must be added to the Portal Load Balancer configuration. Step Use ssh or the vsphere Client console to access the command line on the Portal Load Balancer 0. Edit the /etc/haproxy/haproxy.conf file. listen portal_proxy lbpvip0:80 mode http source usersrc client cookie SERVERID insert nocache indirect server portal0 portal0:80 weight cookie portal0 check server portal0 portal0:80 weight cookie portal0 check Step If another portal03 and portal04 are added, the following lines need to be added to the file: server portal03 portal03:80 weight cookie portal03 check server portal04 portal04:80 weight cookie portal04 check Step 3 To complete the task, run this command from the CLI: service haproxy restart Step 4 Next, perform the same steps on the Portal Load Balancer 0. As new features or fixes to the solution become available, Cisco posts newer software to the Cisco web site for download. Use the Cisco web site to download a.tgz file to your local machine and then upload to the Control Center 0 system into the /var/broadhop/repositories/cisco-sp-wifi directory. Untar it using tar xfz and a compatible file name. 4-3

32 Expanding Architecture for Geographic Redundancy Chapter 4 Expanding the Deployment Expanding Architecture for Geographic Redundancy To handle a complete failure of a QPS cluster or the loss of a data center hosting a QPS, the QPS may be deployed in geo-redundant QPS clusters in which two or more QPS clusters are deployed in multiple sites and linked together for redundancy purposes. QPS supports both active-standby and active-active geo-redundant architectures. In an active-standby model, one site is the designated primary site carrying load (active). The secondary, backup sites are only used in the event of failure of the active site. In an active-active model, all sites within the geo-redundant cluster actively carry traffic, but are dimensioned with sufficient capacity to handle the additional traffic load in the event of a site failure of geographic proportion. In the event of a site failure, the PCEFs need to be treated as follows: They must be configured to automatically detect a site failure and switch to the standby QPS cluster or They must be able to be switched through manual intervention Similarly, it is the responsibility of northbound systems (for example, provisioning systems) to detect site failures and support either switching or reconfiguration to the standby QPS cluster as necessary. Data Synchronization Geo-redundancy is achieved by synchronizing data across the sites in the cluster. Three types of data is replicated across sites: Service and policy rule configurations Subscriber data that are stored in the Quantum Unified SuM component Balance data stored in the Quantum MsBM component In addition, active session data stored in the Session Manager component may also be synchronized across sites when network conditions permit. Active session data is the most volatile data in QPS and has the most stringent synchronization requirements. Session replication across sites has these requirements: Internal QPS VMs in both of the geographically dispersed sites must exist in the same VLAN. Bandwidth must be on a 40MBps link or faster. Latency must be less than 0 milliseconds. Packet loss must be less than 0.%. QPS utilizes a unicast heartbeat between sites in the geographic redundant solution. The heartbeat allows the session manager components to Know which is the currently active component. Protect against a split-logic scenario where data is accepted at more than one session manager component, possibly causing data corruption. An additional external component called an arbiter provides a tie-breaking vote as to which of the session managers is the current master. This external component is required to reside on a separate site from the primary and secondary sites and must be routable from both sites. An arbiter ensures that if one of the sites is lost, the arbiter has the ability to promote the standby site s session manager to be the master. 4-4

33 Chapter 4 Expanding the Deployment Expanding Architecture for Geographic Redundancy To support this architecture, sessionmgr0, sessionmgr0, sessionmgr03, sessionmgr04 and arb0 must be able to route to each other across the sites across the following ports: sessions balance SPR 770 SPR Typically this routing is done over the signaling traffic interface because there are other requirements relating to the Policy Directors that have the same routing requirements. Figure 4- shows a detailed example of the data synchronization architecture for subscriber, balance, and session data. Figure 4- Synchronization Architecture for Subscriber, Balance, And Session Data Site A (Active) Site C Site B (Standby) SM0 Active Sychronization SM0 Standby Sychronization ARB0 Sychronization SM03 Standby Sychronization SM04 Standby In the case of Site A failure, Site B s session manager will become master as shown in Figure 4-. Figure 4- Session Manager Becoming Master Site A (Active) Site C Site B (Standby) SM0 Active Sychronization SM0 Standby Sychronization ARB0 Sychronization SM03 Active Sychronization SM04 Standby 4-5

34 H P PG 6roLiant B L460c H P PG6roLiant B L460c H P PG6roLiant B L460c HP ProLiant G6 HP ProLiant G6 HP P G6 roliant H P PG6 roliant B L460c HP ProLiant G6 HP ProLiant G6 HP ProLiant G6 HP ProLiant G6 HP ProLiant G6 HP ProLiant G6 H P PG6 roliant B L460c HP ProLiant G6 HP ProLiant G6 Expanding Architecture for Geographic Redundancy Chapter 4 Expanding the Deployment Active-Standby Mode In active-standby mode, one QPS system is active while the other QPS system, often referred to as the Disaster Recovery (DR) site, is in standby mode. In the event of a complete failure of the primary QPS cluster or the loss of the data center hosting the QPS, the standby site takes over as the active QPS cluster. All PCEFs use the active QPS system as primary, and have the standby QPS system configured as secondary. The backup QPS system is standby in the sense that it does not receive any requests from connected PCEFs unless the primary QPS system fails, or in the event of a complete loss of the primary site. The backup QPS system is not configured as passive. If your system uses an external load balancer or Diameter Routing Agent (DRA), the QPS in the active cluster is typically configured in one group and the QPS in the standby cluster is configured in a secondary group. The load balancer/dra may then be configured to automatically fail over from active to passive cluster. The active-passive, or Disaster Recovery mode is illustrated in Figure 4-3. Figure 4-3 Active/Passive HA without External Load Balancer QNS Primary Site QNS DR Site Session state synchronisation Radius/CoA Primary Radius/CoA Primary Radius/CoA Secondary Radius/CoA Primary Radius/CoA Secondary Radius/CoA Secondary Cisco ISG Cisco ISG Cisco ISG Active-Active Geographic Clusters The most common QPS geographically redundant architecture supports two or more QPS active-standby clusters deployed in an active-active configuration across multiple PCEFs. Each site contains the active for the local PCEFs and the other site contains the standby for the remote PCEFs. Each PCEF is configured in active-passive mode with one QPS cluster as primary and another QPS cluster as standby. 4-6

35 HP ProLiant G6 HP ProLiant G6 HP ProLiant G6 HP ProLiant G6 HP ProLiant G6 HP PGroLiant 6 HP ProLiant G6 H P PG6 roliant B L460c HP ProLiant G6 HP PG6 roliant H P PG6 roliant B L460c HP ProLiant G6 HP PGroLiant 6 HP ProLiant G6 H P PG6 roliant B L460c HP ProLiant G6 Chapter 4 Expanding the Deployment Expanding Architecture for Geographic Redundancy As such, the PCEF automatically switches over to the standby QPS cluster if the primary QPS cluster fails. The failure detection is supported in the PCEF or NAS device, perhaps as a result of a connection failure or server busy message. In the case of either a complete QPS system failure or the loss of the data center in which the QPS system is installed, the second QPS system will process all requests from all subscribers. If the subscriber session state is synchronized between the sites, the failover is stateful and no sessions need to be re-established. The active-active mode is illustrated in Figure 4-4. Each PCEF is configured with a primary and a secondary RADIUS connection to the deployed QPS systems. Figure 4-4 Active/Active QPS HA Without External Load Balancer QNS Site A QNS Site B Session State Synchronization Radius/CoA Primary Radius/CoA Primary Radius/CoA Secondary Radius/CoA Secondary Radius/CoA Secondary Radius/CoA Primary Cisco ISG Cisco ISG Cisco ISG In the following diagram, Figure 4-5, two external load balancers send traffic to the QPS installations. 4-7

36 H P PG6roLiant B L460c HP ProLiant G 6 HP ProLiant G6 H P PG6roLiant B L460c HP ProLiant G 6 HP ProLiant G6 HP ProLiant G 6 HP ProLiant G 6 HP ProLiant G 6 HP ProLiant G6 HP ProLiant G 6 HP ProLiant G 6 HP PG6 roliant HP ProLiant G6 HP ProLiant G 6 HP ProLiant G 6 Expanding Architecture for Geographic Redundancy Chapter 4 Expanding the Deployment Figure 4-5 Active/Active QPS HA with External Load Balancer QNS Site A QNS Site B Session State Synchronization Radius/CoA Primary Radius/CoA Secondary Radius/CoA Secondary Radius/CoA Primary Radius/CoA Primary Radius/CoA Primary Radius/CoA Primary Cisco ISG Cisco ISG Cisco ISG Session Manager Configuration After the base install has been completed on both sites, the standard Session Manager configuration on each site will be as shown in Figure 4-6. Figure 4-6 Standard Session Manager Configuration at Installation Quantum Server Blade Control Center Blade Quantum Server Blade SM0 Active Sychronization ARB0 Sychronization SM0 Standby Lives on pcrfclient0 Verifying Session Manager Cluster These steps verify the session manager cluster. 4-8

37 Chapter 4 Expanding the Deployment Expanding Architecture for Geographic Redundancy Step To verify that the session manager cluster is up, use the command line interface on sessionmgr0, and run the following commands: [root@sessionmgr0 ~]# /usr/bin/mongo --port 777 MongoDB shell version:..0 connecting to: :777/test set0:primary> rs.status() The result should look like the following code response, where one of the session manager instances is the PRIMARY, one is the SECONDARY, and the arbiter is running on pcrfclient0.xxxx { "set" : "set0", "date" : ISODate("03-0-5T:33:8Z"), "mystate" :, "members" : [ { }, { "_id" : 0, "name" : "sessionmgr0:777", "health" :, "state" :, "statestr" : "SECONDARY", "uptime" : , "optime" : Timestamp( , 8), "optimedate" : ISODate("03-0-5T:33:09Z"), "lastheartbeat" : ISODate("03-0-5T:33:6Z"), "pingms" : 0 "_id" :, "name" : "sessionmgr0:777", "health" :, "state" :, 4-9

38 Expanding Architecture for Geographic Redundancy Chapter 4 Expanding the Deployment "uptime" : , "statestr" : "PRIMARY", "optime" : Timestamp( , 8), "optimedate" : ISODate("03-0-5T:33:09Z"), "self" : true }, { "_id" :, "name" : "pcrfclient0:777", "health" :, "state" : 7, "statestr" : "ARBITER", "uptime" : 78693, "lastheartbeat" : ISODate("03-0-5T:33:6Z"), "pingms" : 0 } ], "ok" : } Step To verify the setup, repeat these steps for the other instances on 778, 779 and 770. Step 3 After verification: Procure five IP routable IP addresses on the Management/Signaling network. In the examples listed in this document, this will be in the /4 IP range. Procure a hostname to be used for external arbiter. Two new hostnames for site B s sessionmgr03 and sessionmgr04. For reference, we will call the new arbiter arbiter0, and the IP will be The new sessionmgr0 and sessionmgr0 IPs in site A will be and The new sessionmgr03 and sessionmgr04 IPs in site B will be and The next steps install the Arbiter on site C hardware. Step Step Start by cloning the pcrfclient0 VM and install it on the new hardware containing VMWare ESX. When this is complete, IP the VM and change the hostname to be arbiter0 in the /etc/hostname file and the /etc/hosts file. 4-0

39 Chapter 4 Expanding the Deployment Expanding Architecture for Geographic Redundancy Step 3 Because this was originally a pcrfclient, stop some of the services on it. To do this, issue these commands: service qns stop chkconfig qns off Step 4 Step 5 Step 6 Step 7 Step 8 Add the management sessionmgr0 and sessionmgr0 and IP the new interfaces in site A. This may require you to restart the VM. See the VMware documentation for specific instructions on how to do this. See G-66B9537-3D-446B-9DCF-6A66E09B0B7.html Add the management sessionmgr0, sessionmgr0, and IP the new interfaces in site B. This addition may require to restart the VM. Add the host name of the arbiter0, sessionmgr03 (originally site B sessionmgr0), sessionmgr04 (originally site B sessionmgr0) and their appropriate IPs to the /etc/hosts of the pcrfclient0 and synch it to all systems. Check all of the configurations by pinging arbiter0, sessionmgr0, sessionmgr0, sessionmgr03 and sessionmgr04 from each of those respective VMs. If all of those can be pinged, the next check is to make sure that all of the ports can be opened from all of the VMs to each other. Run this command: /usr/bin/mongo --port host arbiter0 Step 9 Replace the port for ports 777, 778, 779, 770 and the host with arbiter0, sessionmgr0, sessiongmr0, sessionmgr03, sessionmgr04 from each respective VM. The result should look like this: MongoDB shell version:..0 connecting to: arbiter0:777/test Step 0 Run these commands on pcrfclient0 to stop its mongo instances since they will be running on arbiter0. chkconfig sessionmgr-777 off chkconfig sessionmgr-778 off chkconfig sessionmgr-779 off chkconfig sessionmgr-770 off Step Run these commands from pcrfclient0 to build the new replica sets. build_set.sh set0 /data/sessions. arbiter0:777 sessionmgr0:777 sessionmgr0:777 4-

40 Expanding Architecture for Geographic Redundancy Chapter 4 Expanding the Deployment sessionmgr03:777 sessionmgr04:777 build_set.sh set0 /data/sessions. arbiter0:778 sessionmgr0:778 sessionmgr0:778 sessionmgr03:778 sessionmgr04:778 build_set.sh set03 /data/sessions.3 arbiter0:779 sessionmgr0:779 sessionmgr0:779 sessionmgr03:779 sessionmgr04:779 build_set.sh set04 /data/sessions.4 arbiter0:770 sessionmgr0:770 sessionmgr0:770 sessionmgr03:770 sessionmgr04:770 Step After the replica set is fully online (wait about 0 minutes to be sure or tail the log file or run rs.status() (to check the status), retest the original cluster or if this a new deployment try running some basic tests to confirm all of the connectivity is set. 4-

41 Chapter 4 Expanding the Deployment Expanding Architecture for Geographic Redundancy Policy Director Configuration When an active-active cluster is deployed, both sides have active sessions that will be replicated only to the other cluster as a standby. When back office systems need to make changes to subscribers (SPR) or their balances (MsBM), they will not always know which cluster the subscriber s active session will be on. To facilitate the decision, the QPS system has the ability to broadcast the messages resulting from subscriber or balance changes to other clusters. To configure this feature, you will need the lb0 and lb0 IP addresses of the remote active cluster. For this example, assume these are IPs and Step Step Step 3 Open a command line on pcrfclient0. Add these IPs to the /etc/hosts file as remote-lb0 and remote-lb0. After adding those, find the /etc/broadhop/iomanager0/qns.conf and /etc/broadhop/iomanager0/qns.conf. Find the section in both that looks like this: QNS_OPTS=" -DbrokerUrl=vm://localhost -DjmsFlowControlHost=localhost -DjmsFlowControlPort=9045 -DembeddedBrokerEnabled=true -DembeddedBrokerNetworkUri=static:(tcp://lb0:666)?useExponentialBackOff=false " Step 4 After the DembeddedBrokerNetworkUri property, add a new property like this: -DclusterPeers=failover:(tcp://remote-lb0:666,tcp://remote-lb0:666)?updateURIsSupported =false!cluster-.default The new configuration should look like this: QNS_OPTS=" -DbrokerUrl=vm://localhost -DjmsFlowControlHost=localhost -DjmsFlowControlPort=9045 -DembeddedBrokerEnabled=true -DembeddedBrokerNetworkUri=static:(tcp://lb0:666)?useExponentialBackOff=false -DclusterPeers=failover:(tcp://remote-lb0:666,tcp://remote-lb0:666)?updateURIsSupported =false!cluster-.default 4-3

42 Expanding Architecture for Geographic Redundancy Chapter 4 Expanding the Deployment " Step 5 Step 6 Step 7 After these changes, synch all of the /etc/hosts and configuration files to all of the other VMs and then restart iomanager0 and iomanager0 to get the changes to take effect. After the configuration changes, test by starting a session on the remote cluster and then change the subscribers profile in SPR in the local cluster. The session should be changed in the remote cluster, and any network device changes should be made as well. After confirming the tests, repeat the procedure in the remote cluster. 4-4

43 CHAPTER 5 Quantum Policy Server Blade Hardware Revised: September 30, 03, This section describes the blade hardware used in Quantum Policy Suite, and the number of blades needed to deploy. Specifically, look at these sections for information on what your hardware consists of and how to grow your QPS solution. This chapter covers the following topics: Server Prerequisites 5Blade Server Requirements Server Prerequisites For specific hardware recommendations, see the document Quantum Policy Suite System Sizing Guide, The Quantum Policy Suite can be installed on a variety of non-specific hardware. For example, a deployment site might use blade hardware or an HP DL 380 Generation 7. You can obtain hardware directly from Cisco Systems, Inc. or through your own purchasing department. 5Blade Server Requirements The Quantum Policy Suite is optimized for a commercial off-the-shelf (COTS) blade server and ATCA environment, such as those from IBM, Sun Microsystems, HP, Cisco, and RadiSys (Continuous Computing). Blade Server Minimum Hardware Requirements Processor x Intel Xeon X5650 or higher speed processor Memory Minimum RAM Requirement: 48 GB RAM for less than,000,000 subscribers 7 GB RAM for greater than,000,000 subscribers Storage x 00 GB SSD Drives Supporting Hardware RAID with write backed cache Interconnect Dual Gigabit Ethernet ports 5-

44 Chapter 5 Quantum Policy Server Blade Hardware Blade Server Minimum Hardware Requirements Virtualization Must be listed on the VMware HCL Chassis Minimum Hardware Requirements Device Bays 4 Minimum Interconnect Redundant interconnect Support Power Redundant AC or DC power supplies (as required by Service Provider) Cooling Redundant cooling Support Quantum Network Suite Minimum Requirements Redundant System 4 blade servers: Quantum Control Centers plus Quantum Servers 5-

45 GLOSSARY Revised: September 30, 03, 3G systems 3GPP 4G System 3G is the third generation of mobile phone standards and technology. It is based on the international Telecommunication Union (ITU) family of standards under the International Mobile Telecommunications Programme, MT-000. Third Generation Partnership Project 4G is the fourth generation of cellular wireless standards, a successor to the 3G and G families of standards. In 008, the ITU-R organization specified the IMT-Advanced (International Mobile Telecommunications Advanced) requirements for 4G standards, setting peak speed requirements for 4G service at 00 Mbit/s for high mobility communication (such as from trains and cars) and Gbit/s for low mobility communication (such as pedestrians and stationary users. A AAA AAR ADC ADN AF Command for an AA Answer Command for an AA Request Application Detection and Control Application Delivery Network Application Function Element offering application(s) that use IP bearer resources. NOTE: One example of an AF is the P-CSCF of the IM CN subsystem. AF Session Application-level session established by an application-level signaling protocol offered by the AF that requires a session set-up with explicit session description before the use of the service. Note: One example of an application session is an IMS session. AN Gateway answer service template API application Access Network Gateway A service template for Diameter; answering a request. Application Programming Interface A Diameter application that defines the data needed to perform various related actions. GL-