Cisco Service Control Collection Manager User Guide

Transcription

1 Cisco Service Control Collection Manager User Guide Release 4.0.x November 14, 2013 Cisco Systems, Inc. Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at Text Part Number:

2 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB s public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R) Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental Cisco Systems, Inc. All rights reserved.

3 CONTENTS About this Guide xi Introduction xi Document Revision History xii Organization xiii Related Publications xiv Conventions xv Obtaining Documentation and Submitting a Service Request xvi CHAPTER 1 Cisco Service Control and Cisco Service Control Engine Overview 1-1 Cisco Service Control Solution 1-2 Service Control for Broadband Service Providers 1-2 Cisco Service Control and Cisco SCE Capabilities 1-3 Cisco SCE Platform Description 1-4 Management and Collection 1-6 Network Management 1-6 Subscriber Management 1-7 Service Configuration Management 1-7 Data Collection 1-7 CHAPTER 2 Cisco Service Control Collection Manager Overview 2-1 Data Collection Process 2-2 Raw Data Records 2-2 Collection Manager Software Package 2-5 Raw Data Record Server 2-5 Categorizer 2-5 Priority Queue and Persistent Buffer 2-5 Adapter 2-6 JDBC Adapter 2-6 Comma-Separated Value Adapter 2-6 Custom Adapter 2-7 Operating System Finger Print Reporting 2-7 Topper Aggregator (TA) Adapter 2-8 TA Adapter Cycles 2-8 iii

4 Contents TA Adapter Memory Requirements 2-9 Real-Time Aggregating Adapter 2-11 RAG Adapter Aggregation Buckets 2-11 Flushing a Bucket 2-11 RAG Adapter Process for HTTP Transaction Usage RDR 2-12 RAG Adapter Process for Video Transaction Usage RDR 2-12 RAG Adapter Mobile Based Reports 2-12 RAG Adapter Process for Subscriber Usage RDR 2-12 Managing OSFP support for Subscriber Usage RDR 2-13 Using Databases 2-14 Using the Bundled Database 2-14 Using an External Database 2-15 CHAPTER 3 Installing Collection Manager 3-1 System Requirements 3-2 Supported Platforms 3-2 Checking System Prerequisites 3-2 Hardware Requirements 3-3 Solaris Requirements 3-3 Software and Environment 3-4 Setting the Locale and Time Zone 3-5 Red Hat Linux Requirements 3-6 Software and Environment 3-6 Setting the Locale and Time Zone 3-7 CentOS Linux Requirements 3-7 Distribution Content 3-7 Default Configuration Settings 3-7 How to Install Collection Manager 3-8 Getting Collection Manager Software 3-8 Collection Manager Software Ports 3-9 Installing the Bundled Sybase Database 3-9 Actions the installsyb.sh Script Performs During Installation 3-10 Prerequisites 3-10 Upgrading the Bundled Sybase Database 3-11 Actions the installsyb.sh Script Performs During an Upgrade 3-11 Prerequisites 3-11 Installing the Collection Manager Software 3-12 Information About the install-cm.sh Script 3-12 Configuring the External MySQL Server 3-16 iv

5 Contents Uninstalling the Sybase Database and Collection Manager Software 3-17 Uninstalling Sybase 3-17 Uninstalling Collection Manager Software 3-17 Upgrading Collection Manager to Version 4.0.x 3-18 CHAPTER 4 Managing Collection Manager 4-1 Using Utility Scripts 4-2 Collection Manager Support Information 4-2 Configuring Collection Manager 4-3 Using Scripts to Configure the Collection Manager 4-3 Activating Servers 4-3 Controlling Adapters 4-4 Enabling an Adapter 4-5 Dropping a Cisco SCE Connection 4-5 Configuring a Database 4-5 Configuring the Timezone Offset 4-6 Using Files to Configure the Collection Manager 4-6 Configuring the Categorizer 4-7 Monitoring System Health 4-8 Installing the Periodic Checker 4-8 Periodic Checker Script 4-8 Periodic Checker Script 4-8 Tests 4-9 Managing Users 4-10 Examples 4-10 Managing Virtual Links 4-11 Managing IMEI or MEID TAC with Device Types 4-12 Update the IMEI_DEVICETYPE table 4-12 Update the MEID_DEVICETYPE Table 4-13 Update the OSFP_TYPES Table 4-13 Managing the Peak Hour Configuration 4-14 Monitoring Collection Manager 4-15 Checking the Database Capacity 4-15 Checking the RDR Rate 4-15 Checking the SCE Connection 4-16 Example 4-16 Verifying Server Operation 4-16 Viewing Database Statistics 4-17 v

6 Contents Viewing Version Information 4-17 CHAPTER 5 Managing Databases and the Comma-Separated Value Repository 5-1 Common Database Management Tasks 5-2 Listing Database Tables 5-2 Managing the Periodic Deletion of Old Records 5-3 Configuring Periodic Delete 5-4 Applying the Periodic Delete Configuration File 5-5 Deleting a Table 5-6 Deleting Old Records 5-6 Unicode Support for Sybase Database 5-7 Managing the Bundled Sybase Database 5-8 Backing Up the Database 5-8 Updating Sybase with a Changed IP Address 5-9 Updating Sybase Database Statistics 5-9 Managing the External MySQL Database 5-10 Backing Up the MySQL Database 5-10 Restoring a MySQL Database 5-10 Partitioning the MySQL Database 5-10 Prerequisites to Configuring Partitioning 5-11 Configuring Partitioning 5-11 Managing the Oracle Database 5-13 Oracle Support with Minimum Privileges 5-13 Aggregating Database Data 5-14 Configuring Aggregation 5-15 Enabling Aggregation in Sybase 5-16 Enabling Aggregation in MySQL 5-16 Enabling Aggregation in Oracle 5-17 Managing the CSV Repository 5-18 CSV Repository File Structure 5-18 Configuring the CSV File Repository 5-18 Configuring the Comma Escape 5-20 Configuring Escape of Nonprintable Characters 5-20 CHAPTER 6 Configuring Databases 6-1 Changing Basic Connection Parameters 6-1 Generating SQL Code Using the Velocity Template Language 6-1 Database Configuration Files 6-3 vi

7 Contents Context Objects 6-4 The tables Object 6-4 The dbinfo Object 6-4 The tools Object 6-5 Application Configuration 6-5 Working Sample 6-6 Macro Definitions 6-6 dbinfo Configuration 6-7 SQL Definitions 6-7 Code for Dropping a Table 6-7 Code for Creating a Table 6-7 Code for Creating Indexes 6-8 Code for Insert 6-8 Code for Metadata Query 6-8 Testing and Debugging 6-9 Parsing a String 6-9 Obtaining Complete Debug Information 6-9 Using the JDBC Framework in Scripts 6-10 Viewing and Setting a Cisco SCE Time Zone Offset 6-10 Scalability Hints for Oracle 6-12 Using Custom Tablespaces 6-12 Using Table Partitioning 6-12 APPENDIX A Code Samples A-1 Application Configuration A-2 The dbtables.xml File A-2 The tables.dtd File A-3 RAG Adapter Configuration A-5 The ragadapter.conf File A-5 ragadapter.xml File A-6 The http_turs.xml File A-6 The video_turs.xml File A-7 The vlink_bw_per_pkg.xml File A-8 The vsa_surs.xml File A-8 The cdma_surs.xml file A-9 Custom Adapter Configuration A-10 APPENDIX B Configuration File Descriptions B-1 General Configuration Files B-2 vii

8 Contents Collection Manager Configuration B-2 The [adapter] Section B-2 The [adapter_mem] Section B-3 The [adapter_start_timeout] Section B-3 The [buffer] Section B-4 The [queue] Section B-4 The [categorizer] Section B-4 The [server] Section B-5 The [collector] Section B-5 The [csv_adapter] Section B-5 Queue Configuration B-6 Adapter Configuration Files B-7 Topper/Aggregator (TA) Adapter B-7 [config] Section B-7 The [peak_hours] Section B-8 The [rdr] Section B-9 The [csv] Section B-10 The [state save] Section B-10 The [db] Section B-10 The [app] Section B-11 The CSV Adapter Configuration File B-11 The [csvadapter] Section B-12 [bench] Section B-12 The JDBC Adapter Configuration File B-12 The [bench] Section B-12 The [db] Section B-13 The [app] Section B-13 RAG Adapter Configuration File B-14 The [config] Section B-14 The [housekeeper] Section B-14 The [db] Section B-15 The [app] Section B-15 Comma-Separated Value Adapter B-16 The [customadapter] Section B-16 The [bench] Section B-16 Custom Adapter Configuration File B-17 The [customadapter] section B-17 The [bench] section B-17 The [osfp] section B-17 The [db] section B-18 viii

9 Contents The [app] section B-18 Customadapter Configuration Example B-19 ix

10 Contents x

11 About this Guide Introduction This guide describes how to install and configure the Cisco Service Control Collection Manager as part of the Cisco Service Control Engine (SCE) solution. This guide is intended for technicians who must install and configure the Collection Manager. The guide is also intended for service provider operators who must manage the daily operations of the Collection Manager. Technicians and operators should be familiar with the Cisco Service Control Solution, Cisco Service Control Engine (SCE) platforms, and related components. xi

12 Document Revision History Table 1 lists the changes to this document since it was first released. Table 1 Revision OL Document Revision History Cisco Service Control Release and Date Supports all 4.0.x releases November 04, 2013 Supports all 4.0.x releases June 21, 2013 Change Summary Updated the TA Adapter Memory Requirements, page 2-9. Updated for Release The following sections are updated: RAG Adapter Process for Video Transaction Usage RDR section on page 2-12 The video_turs.xml File section on page A-7 Operating System Finger Print Reporting section on page 2-7 Custom Adapter Configuration File section on page B-17 Collection Manager Configuration section on page B-2 updated with example for the new adapter memory configuration. xii

13 Organization This chapter describes the contents of each chapter in this document. Table 2 Document Organization Section Title Description Chapter 1 Cisco Service Control and Cisco Provides a functional overview of the Cisco Service Control Engine Overview Service Control solution. Chapter 2 Cisco Service Control Collection Describes the Collection Manager components. Manager Overview Chapter 3 Installing Collection Manager Describes how to install the Collection Manager and its database, and explains how to run the Collection Manager. Chapter 4 Chapter 5 Chapter 4, Managing Collection Manager Chapter 5, Managing Databases and the Comma-Separated Value Repository Explains how to use utility scripts to view and update Collection Manager parameters and other information. Explains how to use utility scripts to manage the Collection Manager database and the CSV repository. Chapter 6 Configuring Databases Explains how to configure the Collection Manager to work with your database. Appendix A Appendix B Code Samples Configuration File Descriptions Provides sample lists of code for configuration files. Describes the available parameters in all of the configuration files. xiii

14 Related Publications Use this guide with the following Cisco documentation: Cisco Service Control Application for Broadband User Guide Cisco Service Control Application for Broadband Reference Guide Cisco SCA BB Service Configuration API Programmer Guide Cisco Service Control Application Reporter User Guide xiv

15 Conventions Table 3 describes the syntax used with the commands in this document: Table 3 Conventions Convention bold font italic font Indication Commands and keywords and user-entered text appear in bold font. Document titles, new or emphasized terms, and arguments for which you supply values are in italic font. [ ] Elements in square brackets are optional. {x y z } Required alternative keywords are grouped in braces and separated by vertical bars. [ x y z ] Optional alternative keywords are grouped in brackets and separated by vertical bars. string A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks. courier font Terminal sessions and information the system displays appear in courier font. < > Nonprinting characters such as passwords are in angle brackets. [ ] Default responses to system prompts are in square brackets.!, # An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line. Means reader take note. Tip Means that the following information will help you solve a problem. Caution Means reader be careful. In this situation, you might perform an action that could result in equipment damage or loss of data. Timesaver Means the described action saves time. You can save time by performing the action described in the paragraph. Warning Means reader be warned. In this situation, you might perform an action that could result in bodily injury. xv

16 Obtaining Documentation and Submitting a Service Request For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at: Subscribe to What s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0. xvi

17 CHAPTER 1 Cisco Service Control and Cisco Service Control Engine Overview This chapter presents an overview of the Cisco Service Control solution operating with the Cisco Service Control Engine (SCE). The Collection Manager uses the hardware capabilities of the Cisco SCE platform. It identifies the combination of Cisco-specific applications that create the Cisco Service Control solution. Cisco Service Control Solution, page 1-2 Cisco Service Control and Cisco SCE Capabilities, page 1-3 Cisco SCE Platform Description, page 1-4 Management and Collection, page

18 Cisco Service Control Solution Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Cisco Service Control Solution The Cisco Service Control solution combines hardware and software components. Service providers can use the SCE platform to classify, analyze, and control Internet and IP traffic. Service Control for Broadband Service Providers TheCisco SCA BB adds a layer of service intelligence and control to existing networks that can: Report and analyze network traffic at subscriber and aggregate level for capacity planning Provide customer-intuitive tiered application services and guarantee application service level agreements (SLAs) Implement different service levels for different types of customers, content, or applications Identify network abusers who are violating the acceptable use policy (AUP) Enforce the AUP Identify and manage peer-to-peer traffic, NNTP (news) traffic, and spam abusers Integrate Service Control solutions easily with existing network elements and business support systems (BSS) and operational support systems (OSS) 1-2

19 Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Cisco Service Control and Cisco SCE Capabilities Cisco Service Control and Cisco SCE Capabilities The core capabilities of the SCE platform that supports a wide range of applications for delivering service control solutions, include: Subscriber and application awareness Application-level investigation of IP traffic to understand and control its use down to the subscriber level. Subscriber awareness The ability to map IP flows to a subscriber. The status of each eligible subscriber is maintained. An eligible subscriber transmits traffic through the SCE platform. Subscriber awareness enforces the appropriate policy on subscriber traffic. The Cisco Service Control solution achieves subscriber awareness through dedicated integrations with subscriber management repositories. (DHCP or RADIUS servers are examples of subscriber management repositories.) The Cisco Service Control solution also achieves subscriber awareness by sniffing RADIUS or DHCP traffic. Application awareness The ability to understand and analyze traffic up to the application protocol layer (Layer 7). For application protocols implemented using bundled flows, the SCE platform processes and manages the bundling connection between the flows. Application-layer, stateful, real-time traffic control The ability to perform advanced control functions using application-layer, stateful, real-time traffic transaction processing. This processing requires a highly adaptive protocol and an application-level intelligence. Advanced control functions include granular bandwidth metering and shaping, quota management, and redirection. Programmability The ability to add new protocols and to adapt to new services and applications in the service provider environment. Programmability is achieved by using the Cisco Service Modeling Language (SML). Programmability enables a service provider to deploy new services. Programmability also provides an upgrade path for network, application, or service growth. Back-office integration The ability to integrate with existing third-party systems that are available through the service provider. These systems include provisioning, subscriber repositories, billing systems, and Operations Support Systems (OSS). SCE provides a set of open and documented APIs that support this integration. Scalable high-performance service engines The ability to perform all of the operations described in this section at wire speed. 1-3

20 Cisco SCE Platform Description Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Cisco SCE Platform Description The SCE family of programmable network devices performs application-layer stateful-flow inspection of IP traffic, and controls the traffic based on configurable rules. The SCE platform is a network device that uses ASIC components and reduced instruction set computer (RISC) processors to go beyond packet counting and to go into the contents of network traffic. By providing programmable, stateful inspection of bidirectional traffic flows, and mapping these flows with user ownership, SCE platforms perform real-time classification of network use. This classification is the basis of the SCE platform s advanced traffic-control and bandwidth-shaping functionality. Where most bandwidth shaper functionality ends, the SCE platform provides further control and shaping options, including: Layer 7 stateful wire-speed packet inspection and classification Robust support for more than 600 protocols and applications, including: General HTTP, HTTPS, FTP, Telnet, Network News Transfer Protocol (NNTP), Simple Mail Transfer Protocol (SMTP), Post Office Protocol 3 (POP3), Internet Message Access Protocol (IMAP), Wireless Application Protocol (WAP), and others Peer-to-Peer (P2P) file sharing FastTrack-KazaA, Gnutella, BitTorrent, Winny, Hotline, edonkey, DirectConnect, Piolet, and others P2P VoIP Skype, Skinny, DingoTel, and others Streaming and Multimedia Real Time Streaming Protocol (RTSP), Session Initiation Protocol (SIP), HTTP streaming, Real Time Protocol (RTP), and Real Time Control Protocol (RTCP), and others Programmable system core for flexible reporting and bandwidth control Transparent network and Billing System Support (BSS) and OSS integration into existing networks Subscriber awareness that relates traffic and usage to specific customers 1-4

21 MNG 1 LINK/ 10/100/ ACTIVE 1000 MNG 2 LINK/ 10/100/ ACTIVE 1000 CONSOLE AUX PWR A PWR B STATUS BYPASS LINK RX TX RX MM TX LINK RX TX GBE-1 SUB LINE NET RX MM TX LINK RX TX RX MM TX LINK RX GBE-2 SUB LINE/CASCADE NET TX RX MM TX Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Cisco SCE Platform Description Figure 1-1 illustrates a common deployment of an SCE platform in a network. Figure 1-1 SCE Platform in a Network Corporate Users DSL Provider network Aggregation device Cisco SCE 2000 Series 4xGBE SCE platform Peer network & Internet CMTS

22 MNG 1 MNG 2 LINK/ 10/100/ ACTIVE 1000 LINK/ 10/100/ ACTIVE 1000 CONSOLE AUX PWR A PWR B STATUS BYPASS LINK RX TX LINK RX TX LINK RX TX LINK RX TX RX MM TX GBE-1 SUB LINE NET RX MM TX RX MM TX GBE-2 SUB LINE/CASCADE NET RX MM TX Management and Collection Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Management and Collection The Cisco Service Control solution has a management infrastructure, which includes the following components to manage all aspects of the solution: Network management Subscriber management Service Control management These management interfaces comply with common management standards and integrate with existing OSS infrastructure (Figure 1-2). Figure 1-2 Cisco Service Control Management Infrastructure Network management Service policy and quota management Provisioning system Collection Manager CLI and SNMP XML/RPC Subscriber Manager RDRs Subscriber info DHCP or RADIUS Aggregation device Cisco SCE 2000 Series 4xGBE SCE platform Router Network Management The Cisco Service Control solution provides complete network Fault, Configuration, Accounting, Performance, Security (FCAPS) Management. Two interfaces provide network management: Command-line interface (CLI) Accessible through the Console port or through a Telnet connection, use the CLI for configuration and security functions. SNMP Provides fault management (through SNMP traps) and performance-monitoring functionalities. 1-6

23 Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview Subscriber Management Subscriber Management The Cisco Service Control Application for Broadband (SCA BB) enforces policies on different subscribers and tracks usage on an individual subscriber basis. The Cisco Subscriber Manager is middleware used to bridge between the OSS and the SCE platforms. Subscriber information is stored in the Subscriber Manager database. You can distribute subscriber information between multiple platforms according to the actual subscriber placement. The Subscriber Manager provides subscriber awareness by mapping network IDs to subscriber IDs. It can obtain subscriber information by using dedicated integration modules that integrate with Authentication, Authorization, and Accounting (AAA) devices, such as RADIUS servers or DHCP servers. You can obtain subscriber information in the following ways: Push Mode The Subscriber Manager pushes subscriber information to the SCE platform automatically when a subscriber logs in. Pull Mode The Subscriber Manager sends subscriber information to the SCE platform in response to a query from the SCE platform. Service Configuration Management Service configuration management enables you to configure the general service definitions of a service control application. You create a service configuration file and apply it to an SCE platform. The configuration file contains settings for traffic classification, accounting and reporting, and control. The SCA BB application provides tools that enable you to automate the distribution of these configuration files to SCE platforms. This standards-based approach enables you to manage multiple devices in a large network. Service Control provides a GUI for creating and editing configuration files. Service Control also provides a set of APIs for automating the creation of configuration files. Data Collection Data collection occurs in the Collection Manager from: 1. All analysis and data processing functions of the SCE platform result in the generation of Raw Data Records (RDRs), which the SCE platform forwards using a simple TCP-based protocol and a RDR protocol. 2. The Collection Manager processes the RDRs. 3. The Collection Manager software is an implementation of a collection system that receives RDRs from one or more SCE platforms. It collects these records and processes them in one of its adapters. Each adapter performs a specific action on the RDR. 1-7

24 Data Collection Chapter 1 Cisco Service Control and Cisco Service Control Engine Overview RDRs contain various information and statistics, depending on the configuration of the system. Three main categories of RDRs include: Transaction RDRs Records generated for each transaction. A transaction is a single event detected in network traffic. The identification of a transaction depends on the particular application and protocol. Subscriber Usage RDRs Records generated per subscriber, which describe the traffic generated by that subscriber for a defined interval. Link RDRs Records generated per link, which describe the traffic carried on the link for a defined interval. 1-8

25 CHAPTER 2 Cisco Service Control Collection Manager Overview This chapter is an overview of the Cisco Service Control Collection Manager operating with the Cisco Service Control Engine (SCE). Cisco Service Control Collection Manager uses the hardware capabilities of the SCE platform and identifies the combination of Cisco-specific applications that create the Cisco Service Control solution. This chapter describes the following: How the Cisco Service Control Collection Manager works The Raw Data Records (RDRs) that the SCE platforms generate and send to Collection Manager The components of Collection Manager software package The database used to store the RDRs The chapter includes the following sections: Data Collection Process, page 2-2 Raw Data Records, page 2-2 Collection Manager Software Package, page 2-5 Adapter, page 2-6 Using Databases, page

26 Data Collection Process Chapter 2 Cisco Service Control Collection Manager Overview Data Collection Process The Cisco SCE platform generates RDRs. An application running on the SCE platform, such as the Cisco Service Control Application for Broadband (SCA BB), specifies and defines the RDRs. The following is the data collection process: 1. The SCE platform streams RDRs by using the simple, reliable RDR-Protocol. Integrating the collection of data records with the Service Control Solution involves implementing RDR-Protocol support in the collection system. 2. Collection Manager receives the RDRs from the SCE platform. The software modules recognize and sort the various types of RDRs. The sorting uses preset categories, including type and priority. Collection Manager queues the RDRs in persistent buffers. 3. Collection Manager adapters process each RDR. Each adapter performs a specific function on an RDR: Stores it in a customer adapter formatted file on a local machine Sends it to an RDBMS application Performs custom operations You can use preinstalled utility scripts to customize many of the parameters that influence the behavior of Collection Manager. Raw Data Records The SCE platform generates RDRs. The RDRs, their fields, and their semantics depend on the specific service control protocol (SCP) application. Each RDR type has a unique ID known as an RDR tag. Table 2-1 presents examples of SCP application RDRs: 2-2

27 Chapter 2 Cisco Service Control Collection Manager Overview Raw Data Records Table 2-1 Examples of SCP application RDRs Report Type Description RDRs contain... Periodic Subscriber usage report SCE platforms are subscriber-aware network devices; they can report usage records per subscriber. Subscriber identifiers (such as the OSS subscriber ID) Traffic types (such as HTTP, streaming, or peer-to-peer traffic) Usage counters (such as total upstream and downstream volume). The Periodic Subscriber usage report is necessary for usage-based billing services and for network analysis and capacity planning. The Cisco SCA BB Subscriber Usage RDRs fall under this category. 2-3

28 Raw Data Records Chapter 2 Cisco Service Control Collection Manager Overview Table 2-1 Examples of SCP application RDRs (continued) Report Type Description RDRs contain... Transaction-level report SCE platforms perform stateful tracking of each network transaction conducted on the links on which they are located. Using this statefulness, the SCP tracks several OSI Layer 7 protocols (such as HTTP, RTSP, SIP, or Gnutella) to report on various application-level attributes. Basic Layer 3-4 attributes (such as source IP, destination IP, and port number) Protocol-dependant Layer 7 attributes (such as user-agent, hostname for HTTP, or address of an SMTP mail sender) Generic parameters (such as time of day and transaction duration). These RDRs are important for content-based billing schemes and detailed usage statistics. Cisco SCA BB application Transaction RDRs fall under this category. SCP application activity reports The SCP application can program the SCE platform to perform various actions on network traffic. These actions include blocking transactions, shaping traffic to certain rates and limits, and performing application-level redirections. When such an operation is performed, the SCP application can produce an RDR. Cisco SCA BB application s Breaching RDRs Breaching RDRs are generated when the system changes its active enforcement on a subscriber (because usage exceeded a set quota). Cisco SCA BB application s Blocking RDRs Blocking RDRs are generated when an SCE platform blocks a network transaction (according to the rules contained in the current service configuration). 2-4

29 Chapter 2 Cisco Service Control Collection Manager Overview Collection Manager Software Package Collection Manager Software Package Collection Manager software package contains a group of processing and sorting modules. These modules include the following components: Raw Data Record Server, page 2-5 Categorizer, page 2-5 Priority Queue and Persistent Buffer, page 2-5 Raw Data Record Server A Cisco SCE platform sends RDRs to the Collection Manager. The RDR server adds an arrival timestamp and the ID of the source SCE platform. The RDR server then sends the RDR to the categorizer. Categorizer A categorizer classifies each RDR according to its RDR tag. It determines the destination adapter for the RDR and the priority queue through which the RDR is sent. An RDR can be mapped to more than one adapter. A qualified technician defines the flow in a configuration file, based on user requirements. Priority Queue and Persistent Buffer Each adapter has one or more priority queues; a persistent buffer is assigned to each priority queue. A priority queue queues each RDR according to the RDR s priority level and stores it in a persistent buffer until the adapter processes it. A persistent buffer is a nonvolatile storage area that ensures that the system processes RDRs even in cases of hardware, software, or power failures. 2-5

30 Adapter Chapter 2 Cisco Service Control Collection Manager Overview Adapter Adapters are software modules that transform RDRs to match the requirements of the target system. An adapter distributes the RDRs upon requests from a database. Currently, the following adapters are shipped with the Collection Manager: JDBC Adapter, page 2-6 Comma-Separated Value Adapter, page 2-6 Custom Adapter, page 2-7 Topper Aggregator (TA) Adapter, page 2-8 Real-Time Aggregating Adapter, page 2-11 Some of the adapters send data to the database or write data to CSV files. The structures of the database tables, and the location and structures of these CSV files are described in the Cisco Service Control Application for Broadband Reference Guide. Each adapter has its own configuration file. All the configuration files are similar in structure. For a sample Real-Time Aggregating Adapter (RAG) configuration file, see the The ragadapter.conf File section on page A-5. JDBC Adapter The Java Database Connectivity (JDBC) adapter receives RDRs, processes them, and stores the records in a database. The JDBC adapter is compatible with any JDBC-compliant database server, You can configure the JDBC adapter to use a database operating on a remote machine. The JDBC adapter is preconfigured to support the following databases: Sybase Adaptive Server Enterprise (ASE) 12.5, 15.0, and Oracle 9.2, 10.2, and 11 Disable the recycle bin feature available in Oracle 10 and later versions. You can set the initial value of the recyclebin parameter in the text initialization file init<sid>.ora, for example, recyclebin=off. MySQL 4.1, 5.0, and 5.1 Comma-Separated Value Adapter The comma-separated-value (CSV) adapter receives RDRs, processes them, and writes the records to the files on the disk in the CSV format. Enhanced accounting and network traffic analysis records are generated from the records. The service provider OSS can retrieves these records using standard mechanisms. A third-party billing system can also retrieve these records using standard mechanisms. FTP is an example of a standard retrieval method. 2-6

31 Chapter 2 Cisco Service Control Collection Manager Overview Custom Adapter Custom Adapter The Custom Adapter allows you to configure the RDR format. The Custom Adapter receives records with specified fields from an RDR. You can configure only a few fields of TUR. Some of the configurable fields are Sub-ID, Upstream + Downstream, and Application. The corresponding RDR is processed with those specific fields and stored in a CSV file. The Custom Adapter also supports basic arithmetic operations (add, sub, mul) on a few of the RDR fields. The Custom Adapter process is disabled by default. Edit the ~scmscm/cm/config/cm.conf file in the Collection Manager and restart Collection Manager to enable the Custom Adapter. For more information on enabling Custom Adapter, see the Enabling an Adapter section on page 4-5. Operating System Finger Print Reporting The Operating System Finger Print (OSFP) reports contain information about the operating system used by all subscribers on a daily basis. The RPT_OSFP table in the Cisco Service Control Collection Manager database is updated with the operating system usage information for each subscriber. NAT environment and non-nat environment details are updated separately. The value of the NAT_ENV field in the RPT_OSFP table decides whether the operating system was detected in a NAT environment. The NAT environment value is set as 1 if multiple OS is detected for the same subscriber, and the value is set to 0 if the subscriber is using a single operating system. By default, the OSFP Reports feature is disabled. Before enabling the OSFP in Cisco Service Control Collection Manager, enable the OSFP feature in Cisco SCE devices. For details on enabling the OSFP feature using Cisco SCA BB console, see the Cisco Service Control Application for Broadband User Guide. To enable the OSFP Reports in Cisco Service Control Collection Manager: Step 1 Step 2 Enable the custom adapter. To enable the custom adapter, uncomment the customer adapter field #adapter.5=com.cisco.scmscm.adapters.custom.customadapter under the [adapter] section in ~scmscm/cm/config/cm.conf file. Update the value of the active field under the [osfp] section in ~scmscm/cm/config/customadapter.conf file to true. Example of the [osfp] section of the Custom Adapter configuration: [osfp] active=false acc_period=

32 Topper Aggregator (TA) Adapter Chapter 2 Cisco Service Control Collection Manager Overview Topper Aggregator (TA) Adapter The TA Adapter receives subscriber usage RDRs (NURs) and aggregates the data. The TA Adapter outputs Top Reports to the database in CSV format. Top Reports are lists of top subscribers for different metrics. The top 500 volume or session consumers in the last hour is an example of a Top Report metric. The TA adapter also sends the aggregated daily statistics of all the subscribers to the database. The TA Adapter maintains a persistent saved state (saved to disk) to minimize data loss in case of failure. The TA Adapter uses the JDBC adapter infrastructure. The TA Adapter operates with either local or remote JDBC-compliant databases. TA Adapter supports IP-based aggregation along with the existing service-based or package-based aggregation.the service and IP type-based aggregation is enabled by default. The TA Adapter aggregates the records and lists the top subscribers for each combination of service and IP type with a different metric. The package-based aggregation is disabled by default. Update the agg_pkg_level property as true in the ~scmscm/cm/config/taadapter.conf file and restart Collection Manager to enable the package-based aggregation. The TA Adapter information is aggregated locally on a Collection Manager server. We reccomend not to use the Collection Managers with a single database. This may affect the accuracy of TA Adapter information. A TA Adapter file is in the CSV format and populated with the RDR type NUR. The TA adapter works only with subscriber usage RDRs (NURs) and not with real-time subscriber usage RDRs (SURs). TA Adapter Cycles The TA Adapter has the following specifications: TA Adapter Cycles, page 2-8 TA Adapter Memory Requirements, page 2-9 The TA Adapter works in three cycles: short, long, and peak hours (specific hour range). Cycles are fixed intervals. This data is temporarily stored in a persistent buffer directory with a default location of ~scmscm/cm/adapters/taadapter. At the end of an interval, the adapter outputs its aggregated information to the database and to a CSV file. The default interval for the short cycle is 1 hour. The default interval for the long cycle is 24 hours (every day at midnight). You can configure an interval (defined in minutes) and its start and end time. Configure the interval for the peak hours cycle. At the end of the peak hours, the adapter aggregates and outputs details of the top subscribers to the database. The long-cycle interval must be a multiple of the short-cycle interval. The activities in each cycle differ slightly, as follows: Short cycle At the end of each short cycle, the adapter: Adds the aggregated Top Reports of the cycle to the short cycle database table Saves the current state file in case of power failure 2-8

33 Chapter 2 Cisco Service Control Collection Manager Overview Topper Aggregator (TA) Adapter Long cycle At the end of each long cycle, the adapter: Adds the aggregated Top Reports of the cycle to the long cycle database table Saves the current state file in case of power failure Creates a CSV file to contain the aggregated statistics for the long-cycle period Peak-hour cycle At the end of each peak-hour cycle, the adapter: Adds the aggregated Top Reports of the cycle to the peak hour cycle database table Saves the current state file in case of power failure The long cycle data is saved every hour. The save file has 0 to 23:00 hours of data if a power shutdown occurs between 23:00 hrs and 24:00 hrs. The CSV file contains aggregated data. The current state is saved in a nonreadable format. TA Adapter Memory Requirements The TA Adapter requires sufficient dedicated memory. Configure the value of the memory in the cm.conf file in the following location: [adapter_mem] com.cisco.scmscm.adapters.topper.taadapter=<memory for TA Adapter> To calculate the recommended amount of memory for the TA Adapter for a 32 bit operating system, use the following formula: Memory (MB) = (3 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES ) + (32 * NUM_OF_SERVICES) )) /(1024 * 1024)) * (NUM_OF_IPTYPES) To calculate the recommended amount of memory for the TA Adapter for a 64 bit operating system, use the following formula: Memory (MB) = (5.25 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES ) + (32 * NUM_OF_SERVICES) )) /(1024 * 1024)) * (NUM_OF_IPTYPES) To calculate the recommended amount of memory for the TA Adapter with Package Enabled Aggregation for a 32 bit operating system, use the following formula: Memory (MB) = (3 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + ((20 * NUM_OF_PKGS) +48) + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES)+ (32 * NUM_OF_SERVICES) ))) /(1024 * 1024) * (NUM_OF_PKGS ) * (NUM_OF_IPTYPES) To calculate the recommended amount of memory for the TA Adapter with Package Enabled Aggregation for a 64 bit operating system, use the following formula: Memory (MB) = (5.25 * TOTAL_SUBSCRIBERS * ((AVG_SUBS_ID_LENGTH + ((20 * NUM_OF_PKGS) +48) + (2 * NUM_OF_PERIODS * NUM_OF_SERVICES * NUM_OF_IPTYPES)+ (32 * NUM_OF_SERVICES) ))) /(1024 * 1024) * (NUM_OF_PKGS ) * (NUM_OF_IPTYPES) The definitions of the fields in the formula are: TOTAL_SUBSCRIBERS is the total number of subscribers across the network for each aggregation period. AVG_SUBS_ID_LENGTH is the average character length of a subscriber. NUM_OF_PERIODS = Total number of Aggregation cycles. Default value is

34 Topper Aggregator (TA) Adapter Chapter 2 Cisco Service Control Collection Manager Overview NUM_OF_IPTYPES = Total no of ip types supported +1 NUM_OF_SERVICES = Total number of active services + 1 NUM_OF_PKGS = Maximum number of packages for a particular subscriber +1 Cisco Service Control Collection Manager requires 75 percentage additional memory on a 64 bit machine when compared to a 32 bit machine. The 64 bit machine consumes 64 bit of memory to store each memory address pointers for the java application process. This is twice the memory required to store the addresses in a 32 bit machine. This doubles up the memory requirement while using the Cisco Service Control Collection Manager on 64 bit machine. The Package based aggregation and IP Type based aggregation functions of Cisco Service Control Collection Manager consumes more memory to store data and uses more object references that uses 64 bit to store each address in the memory. For Solaris JRE 64 bit, you can set higher values for the configured memory. To configure the TA or RAG adapters to run with the JRE 64 bit, see the The [adapter_mem] Section section on page B

35 Chapter 2 Cisco Service Control Collection Manager Overview Real-Time Aggregating Adapter Real-Time Aggregating Adapter The RAG adapter processes RDRs of one or more types and aggregates the data from predesignated field positions into buckets. The contents of the buckets are written to CSV files. The RAG adapter has the following components and processes: RAG Adapter Aggregation Buckets, page 2-11 Flushing a Bucket, page 2-11 RAG Adapter Aggregation Buckets Flushing a Bucket RAG Adapter Process for HTTP Transaction Usage RDR, page 2-12 RAG Adapter Process for Video Transaction Usage RDR, page 2-12 RAG Adapter Process for Subscriber Usage RDR, page 2-12 Managing OSFP support for Subscriber Usage RDR, page 2-13 A RAG adapter aggregation bucket is indexed by combining values from fields in the RDR. The indexing relation can be one-to-one or many-to-one. The values in the bucket-identifying fields are processed using closures (equivalence classes), which are configured per type of RDR. Bucket-identifying field = field number 3 Closures: 4 = 4,5,6; 10 = 8,10,11 Value in field 3 = 4, 5, or 6; field reported as 4 Value in field 3 = 8, 10, or 11; field reported as 10 You can configure the RAG adapter to monitor the values in certain fields for change relative to the values in the first RDR that entered the bucket. For each monitored field, an action is performed when a value change is detected. The supported actions are: Checkpoint the bucket without aggregating this RDR into it, and start a new bucket with this RDR. Issue a warning to the user log. Buckets, closures, triggers, and trigger actions are defined in an XML file. For a sample XML file, see the ragadapter.xml File section on page A-6. When a bucket is flushed, it is written as one line to a CSV file. The trigger for flushing a bucket (a checkpoint) is the earliest occurrence of any of the following: Time elapsed since the creation of a bucket reaches a configured duration Volume in an accumulated field in a bucket exceeds a configured amount A RAG Adapter, or the entire Collection Manager, goes down An RDR arrives at the bucket with some new value (relative to the bucket contents) in some field The trigger to close a CSV file is the earliest occurrence of one of the following: Time elapsed since the creation of a file has reached a set duration Number of lines in a file has reached a set number A RAG Adapter, or the entire Collection Manager, goes down 2-11

36 Real-Time Aggregating Adapter Chapter 2 Cisco Service Control Collection Manager Overview RAG Adapter Process for HTTP Transaction Usage RDR If a corresponding RDR TAG is configured under the RAG adapter section in the queue.conf file before Collection Manager is started, the RAG adapter processes the HTTP_TUR RDRs. An XML file specific to HTTP_TUR is included in Collection Manager distribution for the RAG adapter in order to manage the HTTP_TUR RDR fields. The aggregation period for processing the RDRs is specified in the http_turs.xml file. Aggregation is based on the domain, package, and service for the corresponding HTTP Transaction Usage RDR. At the end of the aggregation period, the adapter adds the aggregated Top Reports for hosts to the RPT_TOP_HTTP_HOSTS database table. The adapter adds the aggregated Top Reports for domains to the RPT_TOP_HTTP_DOMAINS database table. RAG Adapter Process for Video Transaction Usage RDR If you configure a corresponding RDR TAG under the RAG adapter section in queue.conf file before you start Collection Manager, the RAG adapter processes VIDEO_TUR RDRs. Collection Manager distribution includes an XML file specifically for VIDEO_TUR, which enables the RAG adapter to manage VIDEO_TUR RDR fields. The aggregation period for processing the RDRs is specified in the video_turs.xml file. Aggregation is based on the domain, package, and service for the corresponding VIDEO Transaction Usage RDR. At the end of the aggregation period, the adapter adds the aggregated Top Reports for hosts to the RPT_TOP_VIDEO_HOSTS database table. Also, the adapter adds the aggregated Top Reports for domains to the RPT_TOP_VIDEO_DOMAINS database tables. Starting from Cisco Service Control Collection Manager Release 4.0.0, BIT_RATE is calculated from the Downstream volume and the Duration of the Video TURs. The accumulated bit rate (BIT_RATE) and the corresponding rank (BIT_RATE_RANK) is added to the RPT_VIDEO_HOSTS table and RPT_VIDEO_DOMAINS table for each service and package combination. RAG Adapter Mobile Based Reports Collection Manager supports both GSM and CDMA mobile-based reports. GSM reports are enabled by default. The ~scmscm/cm/config/ragadapter.conf file property (vsa_type=gsm or cdma) configures the mobile report type processed. A user should to place the related XML files in ~scmscm/cm/config/ragadapter for the Collection Manager. By default all the XML files are available in the ~scmscm/cm/config/ragadapter/repository directory. RAG Adapter Process for Subscriber Usage RDR If a corresponding RDR TAG is configured under the RAG adapter section in the queue.conf file before Collection Manager is started, the RAG adapter processes the NUR RDRs. The Subscriber Usage RDRs are processed in two ways: Aggregation Based on VSA Fields, page 2-13 Aggregation Based on Package, page

37 Chapter 2 Cisco Service Control Collection Manager Overview Real-Time Aggregating Adapter Aggregation Based on VSA Fields VSA field aggregation supports either GSM or CDMA mobile-based reports: Collection Manager supports one of the mobile-based reports (either GSM or CDMA). By default, GSM reports are enabled. In the ~scmscm/cm/config/ragadapter.conf file, a new property (VSA_Type = gsm or cdma) is added to configure which mobile report Collection Manager needs to process. During restart, based on the vsa_type configuration, the Collection Manager will copy the corresponding xml config file to the ~scmscm/cm/config/ragadapter file for processing. Aggregation Based on GSM-Specific VSA Fields The vsa_surs.xml file manages GSM-specific VSA fields in the Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file. Aggregation is based on the GSM-specific VSA fields (APN, SGSN, NETWORK_TYPE, DEVICE_TYPE, and USER_LOCATION). At the end of the aggregation period, the adapter adds the aggregated Top Reports related to the GSM-specific VSA fields to their corresponding database tables: RPT_TOP_APN RPT_TOP_SGSN RPT_TOP_NETWORK_TYPE RPT_TOP_DEVICE_TYPE RPT_TOP_USER_LOCATION Aggregation Based on CDMA-Specific VSA Fields Aggregation Based on Package The cdma_surs.xml file manages the CDMA-specific VSA fields in the Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file. Aggregation is based on the CDMA-specific VSA fields (PCF, HOME AGENT, MEID). At the end of the aggregation period, the adapter adds the aggregated Top Reports related to the CDMA-specific VSA fields to their corresponding database tables: RPT_TOP_PCF RPT_TOP_HOME_ AGENT RPT_TOP_MEID The vlink_bw_per_pkg.xml file manages upstream and downstream VLINKs in the Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file. Aggregation is based on the up and down VLINK fields. At the end of the aggregation period, the adapter adds the aggregated Top Reports for the up VLINKS to the RPT_UVLINK database table. Also, the adapter adds the aggregated Top Reports for the down VLINKs to the RPT_DVLINK database table. Rag Adapter package aggregation happens based on PACKAGE_ID, UP_VLINK_ID/DOWN_VLINK_ID, and IP_TYPE. Managing OSFP support for Subscriber Usage RDR Operating System Finger Printing (OSFP) is supported in Collection Manager software. The ~scmscm/script/updatevsaindex.sh script supports backwards compatibilty. 2-13

38 Using Databases Chapter 2 Cisco Service Control Collection Manager Overview The following properties are added in the ~scmscm/cm/config/ragadpater.conf file: attr_index Index position of attribute indicator field in NUR RDR in current the Cisco SCOS version attr_shift_pos Number of fields to be shifted Check the VSA Attribute Indicator field index value for the current Cisco SCOS version. This is based on the Cisco SCOS version update attr_index, attr_shift_pos properties in ~scmscm/cm/config/ragadpater.conf file. The RAG Adapter Configuration File section on page B-14 details the RAG adapter configuration file ragadapter.conf. Then execute the script ~scmscm/script/updatevsaindex.sh script. The script updates the corresponding XML file (vsa_surs.xml/cdma_surs.xml). Table 2-2 shows the VSA Attribute Indicator field index values. Table 2-2 VSA Attribute Indicator Field Index Values Cisco SCOS Version attr_index attr_shift_pos Using Databases Collection Manager can use either a bundled database or an external database to store the RDRs supplied by the SCE platforms. The following sections describe the procedures: Using the Bundled Database, page 2-14 Using an External Database, page 2-15 Using the Bundled Database In the bundled mode, Collection Manager uses the Sybase Adaptive Server Enterprise database. This database enables you to do the following: Support transaction-intensive enterprise applications Store and retrieve information online Warehouse needed information by putting it in a central depository The Sybase database is located on the same server as the other Collection Manager components. It uses a simple schema that includes a group of small, simple tables. 1. The JDBC adapter sends the converted RDRs to the database, where the RDRs are stored in the tables of the simple schema. 2-14

39 Chapter 2 Cisco Service Control Collection Manager Overview Using an External Database 2. You can access these records by using standard database query and reporting tools. (Cisco provides a template-based reporting tool that can generate reports on subscriber usage, network resource analysis, and traffic analysis. For information about the Service Control reporting tool, see the Cisco Service Control Application Reporter User Guide.) You can maintain the database by using operating system commands and scripts. Collection Manager supports automatic purging of old records from the bundled database. By default, Collection Manager automatically purges the report table of every record that is older than two weeks and polls the records once every hour. You can configure database maintenance by using the dbperiodic.sh utility script. For more information on database maintenance, see the Managing the Periodic Deletion of Old Records section on page 5-3. Using an External Database With the JDBC adapter, you can use any JDBC-compliant database (for example, Oracle or MySQL). The database can be either local or remote. To use an external database, perform the following tasks: Step 1 Step 2 Step 3 Configure the JDBC adapter to use this database. Configure a database pack to supply Collection Manager with the parameters of the database (such as its IP address and port). Supply a JDBC driver for the database so that the adapter can connect to the database. For details about configuring Collection Manager to work with an external database, see Chapter 5, Managing Databases and the Comma-Separated Value Repository. 2-15

40 Using an External Database Chapter 2 Cisco Service Control Collection Manager Overview 2-16

41 CHAPTER 3 Installing Collection Manager This chapter describes how to install Collection Manager and the connected database. The chapter also explains how to run Collection Manager. For a typical installation and configuration, see the Collection Manager Quick Start Guide. This chapter includes the following sections: System Requirements, page 3-2 How to Install Collection Manager, page 3-8 Configuring the External MySQL Server, page 3-16 Uninstalling the Sybase Database and Collection Manager Software, page 3-17 Upgrading Collection Manager to Version 4.0.x, page

42 System Requirements Chapter 3 Installing Collection Manager System Requirements The Collection Manager has the following system requirements: Supported Platforms, page 3-2 Checking System Prerequisites, page 3-2 Hardware Requirements, page 3-3 Solaris Requirements, page 3-3 Red Hat Linux Requirements, page 3-6 CentOS Linux Requirements, page 3-7 Distribution Content, page 3-7 Default Configuration Settings, page 3-7 Supported Platforms You can install Collection Manager on any of the following platforms: Sun SPARC machine (64- bit) running 64-bit versions of Solaris 10. (See the Solaris Requirements section on page 3-3.) Intel machine (64-bit) running 64-bit versions of Red Hat Enterprise Linux 5.x, Red Hat Enterprise Linux 6.0, Red Hat Enterprise Linux 6.1, and Red Hat Enterprise Linux 6.2. (See the Red Hat Linux Requirements section on page 3-6.) Intel machine (64-bit) running 64-bit versions of CentOS 5.x. (See the CentOS Linux Requirements section on page 3-7.) Intel machine (64-bit) running VMware Server or VMware ESX VMware-VMvisor-Installer x86_64.iso. Any Cisco Unified Computing System All the configurations use a 32-bit Java Virtual Machine (JVM). Caution Collection Manager must run on a dedicated server. The local database for Collection Manager is also located on that server. You cannot run the Cisco Subscriber Manager or other software applications on the same server that contains the database. When using the bundled Sybase database, the server on which you install Collection Manager can have a maximum of 4 CPU cores. Checking System Prerequisites The Collection Manager distribution contains a script, check_prerequisites.sh, located in the install_scripts directory. The script helps to determine if a system meets the requirements for installing a Collection Manager and the bundled Sybase database. check_prerequisites.sh [ --sybhome=sybhome ] [ --cmhome=cmhome ] [ --datadir=datadir ] 3-2

43 Chapter 3 Installing Collection Manager Hardware Requirements Table 3-1 provides a list of the check_prerequisites.sh script options. Table 3-1 check_prerequisites.sh Script Options --sybhome=sybhome --datadir=datadir --cmhome=cmhome Intended home directory for Sybase installation Intended data directory for Sybase data files (for the Datadir installation method) Intended home directory for Collection Manager installation The main prerequisites checked are: CPU speed Amount of RAM Operating System version (Solaris 10, Red Hat Enterprise Linux 5 or 6) Additional required and optional packages Free space for the Collection Manager and the Sybase database Names for all network interface cards (NICs) Sybase kernel parameters Locale and time zone formats Number of huge pages required by Sybase on Red Hat Enterprise Linux. Hardware Requirements The minimum hardware requirements for Collection Manager are: 500-MHz CPU One GB RAM per CPU Hard disk: One hard disk (18 GB minimum) For bundled installations, a second hard disk of at least 30 GB is recommended to store Sybase data. 100BASE-T network interface The minimum hardware requirements apply only to small deployments. Medium or large deployments have higher hardware requirements. Your network design expert is the best source of information on the required hardware for a particular deployment. Solaris Requirements Install Collection Manager Release or later on Sun SPARC Machines: Software and Environment, page

44 Solaris Requirements Chapter 3 Installing Collection Manager Setting the Locale and Time Zone, page 3-5 Software and Environment Collection Manager requires the following software version when running on Solaris: Solaris 10 Patch level 10 is recommended Solaris Core Installation. Install the additional packages, as shown in Table 3-2. Table 3-2 Additional Packages system SUNWbash GNU Bourne-Again shell (bash) system SUNWgzip The GNU Zip (gzip) compression utility system SUNWzip The Info-Zip (zip) compression utility system SUNWlibC Sun Workshop Compilers Bundled libc system SUNWlibCx Sun WorkShop Bundled 64-bit libc If you are installing Collection Manager in bundled mode with the Sybase database, install the package shown in Table 3-3. Table 3-3 SUNWipc Package system SUNWipc Interprocess Communication (Optional) You can install the packages listed in Table 3-4 (for sysadmin applications such as sys-unconfig). Table 3-4 Optional Packages system SUNWadmap System administration applications system SUNWadmc System administration core libraries You can download these packages from The root (/) partition must have at least 104 MB of free space to install these packages. At least 8 GB free on the partition for Collection Manager. (This memory is used for CSV storage and persistent buffers.) At least 3 GB free on one partition for the Sybase home directory (for installations with bundled Sybase). For bundled installations, a second hard disk of at least 30 GB is recommended to store Sybase data. 3-4

45 Chapter 3 Installing Collection Manager Solaris Requirements Free space on one partition to hold the desired size of the Sybase data and logs (for installations with bundled Sybase). You can configure memory size during installation. The FTP server requires listening to port 21 so that the SCA Reporter can authenticate the server. All the IP addresses configured for the machine NICs require hostnames associated with them in the /etc/hosts or in another active naming service. The requirement is for installations with the bundled Sybase database. Verification is required for an installation due to a limitation of the Sybase Adaptive Server Enterprise. Use the set_shmmax.sh script to configure the kernel memory for installations with the bundled Sybase. The script is located under install-scripts/. At startup,you must load the IPC module by inserting the following lines in the file /etc/system: forceload: sys/shmsys If you are using database periodic delete, enable the scmscm user to schedule and run cron jobs. Setting the Locale and Time Zone For correct Collection Manager and Sybase operation, use the locale U.S. English. Add the following line in the /etc/timezone configuration file to set the locale: LANG=en_US Restart Collection Manager to change to this configuration. To use the U.S. English locale, Collection Manager must be running on the Solaris operating system. Verify that the locale is installed by ensuring that the directory /usr/lib/locale/en_us exists. If the directory does not exist, install the locale files from the Solaris CDs. Setting the OS time zone as an offset from GMT in POSIX format is not recommended, and can lead to problems. Set the time zone in the /etc/timezone configuration file by country (supported) name, as in the following example. TZ=Japan Verify that the country name is supported as a time zone setting by ensuring that it is listed in the directory /usr/share/lib/zoneinfo. If you must use GMT offset, use the zoneinfo format by prepending the prefix :Etc/, as in the following example: TZ=:Etc/GMT+5 3-5

46 Red Hat Linux Requirements Chapter 3 Installing Collection Manager Red Hat Linux Requirements Software and Environment Installing Red Hat Linux The following information applies to running the Collection Manager using Red Hat Linux. Software and Environment, page 3-6 Setting the Locale and Time Zone, page 3-7 Collection Manager Version requires the following software when running on Red Hat Linux: Red Hat Linux 5.x: kernel el5 glibc compat-libstdc Minimum patch level required Update 5.8 and earlier. Red Hat Enterprise Linux 6.x For Red Hat Linux installations, perform the following steps: Apply the latest recommended patches from Red Hat. Reserve at least 8 GB on the partition where you want to install Collection Manager. Collection Manager uses this disk space for CSV storage and persistent buffers. If you are using database periodic delete, enable the scmscm user to schedule and run cron jobs. If you use Red Hat Enterprise Linux 6.x, confirm that the desired database is compatible with Red Hat Enterprise Linux 6.x. Red Hat Enterprise Linux 6.x does not support the Collection Manager bundled Sybase database. Installing Red Hat Linux and the Sybase Database Bundled with the Collection Manager Red Hat Enterprise Linux 6.x does not support the Sybase database bundled with the Collection Manager. For installations with the Sybase database bundled with the Collection Manager, perform the following steps: Install the compat-libstdc++ package. This package is available on the Red Hat installation CD. Install the current Sybase patches. Reserve at least 1 GB on one of the partitions for the Sybase home directory. Verify that all the configured IP addresses for the machine NICs have hostnames associated with them in the /etc/hosts or in another active naming service. This action is required due to a limitation of Sybase Adaptive Server Enterprise. Use the set_shmmax.sh script (located under the install-scripts/) to configure the kernel memory. 3-6

47 Chapter 3 Installing Collection Manager CentOS Linux Requirements Setting the Locale and Time Zone For correct Collection Manager and Sybase operation, use the locale U.S. English (en_us). CentOS Linux Requirements Install Collection Manager Version on a 64 bit machine. Collection Manager requires the following software version when running on CentOS: CentOS Linux 5.x: kernel el5 glibc compat-libstdc CentOS is supported only on Collection Manager software Release and later. Distribution Content Collection Manager installation kit contains scripts for installing Collection Manager and the Sybase database. It also contains: Scripts to support file gathering Scripts for periodic Sybase maintenance Default Configuration Settings Configure the settings for Collection Manager during installation. These settings include: Adapters to enable and their locations Priority Queue parameters Target adapters for each type of RDR (by RDR tag value) Various logging policies Permit only qualified personnel to change these settings. 3-7

48 How to Install Collection Manager Chapter 3 Installing Collection Manager How to Install Collection Manager This section describes how to install Collection Manager Version or later and the Sybase database: Getting Collection Manager Software, page 3-8 Collection Manager Software Ports, page 3-9 Installing the Bundled Sybase Database, page 3-9 Installing the Collection Manager Software, page 3-12 You can install the Collection Manager on Solaris, Red Hat Linux, or CentOS Linux. Getting Collection Manager Software To download Collection Manager software: Step 1 Go to to the Cisco Product/Technology Support Download Software page for the Cisco Service Control Collection Manager at: &release=4.0.0&relind=AVAILABLE&rellifecycle=&reltype=latest Optionally, you can go to the same Cisco Service Control Collection Manager software download page from Cisco.com by navigating to Cisco Product/Technology Support > Download Software > Service Exchange > Cisco Service Control > Cisco Service Control Management Suite > Cisco Service Control Collection Manager. Step 2 Step 3 Step 4 Step 5 Enter your Cisco CCO password when prompted. Download the SCMS-CM Installation software package for your specific operating system. Specific software packages may consist of a single download or consecutive multiple downloads. If there is a single file package, go to Step 5. If there are consecutive multiple downloads, move the downloaded files to the target machine and join them into a single.tar file. For example: # cat cm_part1 cm_part2 >/usr/tmp/cm_full_package.tar Extract the complete package into a temporary directory. For example: # mkdir /usr/tmp/cm_install_temp # cd /usr/tmp/cm_install_temp # tar xvf../cm_full_package.tar 3-8

49 Chapter 3 Installing Collection Manager Collection Manager Software Ports Collection Manager Software Ports Table 3-5 describes the TCP/UDP ports on which Collection Manager software and associated components (such as the Sybase database) listen. This table helps a network administrator to understand the behavior of the software and its adherence to security policies. Table 3-5 Ports on Which Collection Manager Listens Constantly Port Number Description The SCE devices use this port to send RDRs for data collection Internal collection manager. Access is required only from the local machine; external access can be blocked HTTP technician interface Sybase database connectivity through ODBC or JDBC. Required for access to the database (for installations with bundled Sybase) RMI. Used as the management interface between the data collector and the Service Control management server FTP server of Collection Manager. FTP transactions can listen on other ports (22001 to 22100) for data transfer. The protocol negotiates the port number Internal logging of the management user log. Access is required only from the local machine; external access can be blocked The Cisco SCA BB console uses this port to send symbol definitions (values.ini) to Collection Manager. The device listens constantly on the ports in the table. If you cannot enable access on these port numbers, certain operations can fail. Some operations (such as file transfer) cause a device to temporarily open ports other than the listed ports. However, these ports close automatically when the operation that opens additional ports ends. Installing the Bundled Sybase Database If you do not want to install Sybase (for example, when working in unbundled mode), see the Installing the Collection Manager Software section on page Sybase is supported only on Red Hat Enterprise Linux 5.x, CentOS 5.x, and Solaris 10. The maximum size for the bundled Sybase database is 50 GB. Use an external database for databases over 50 GB. 3-9

50 Installing the Bundled Sybase Database Chapter 3 Installing Collection Manager Installing the Sybase database can take as many as 3 hours. You can install Collection Manager with the bundled Sybase database on a server with a maximum of four CPU cores. This is a limitation of the Sybase license. During installation, if you want to reverse the Sybase installation actions (for example, if an installation is interrupted because of a power failure), do the following: 1. Log in as the root user. 2. Run the /install-scripts/uninstall.sh --sybase [datadir] script. The optional [datadir] option removes the sybase data files. Actions the installsyb.sh Script Performs During Installation The installsyb.sh script installs the Sybase database. The script performs the following actions: Verifies the shmem setting for Sybase in the /etc/system (for Solaris) or the /etc/sysctl.conf file (for Red Hat Linux). If the setting is not there, the script inserts it and reboots (after prompting you). Adds a user Sybase and group Sybase. Runs the Sybase installer for your platform. Builds a Sybase server that includes Sybase users and passwords. Starts Sybase. Runs SQL scripts to create Collection Manager database structure. This process is lengthy and requires restarting Sybase several times. Prerequisites Log in as the root user and make the distribution kit contents available on your system or local network. Step 1 Change directory to Sybase in the distribution kit root. Before starting the upgrade, make sure 1 GB of free memory is available for the Sybase. Step 2 Run the installsyb.sh script. installsyb.sh --sybhome=sybhome {-datadir=datadir} SYBHOME is the home directory of the Sybase user (have 4 GB free for Solaris and 3 GB free for Linux) Select one of the following data location options: Specify --datadir=datadir, where DATADIR is a directory in which to store all Sybase data. Use a location in a partition in which at least 15 GB is free. 3-10

51 Chapter 3 Installing Collection Manager Upgrading the Bundled Sybase Database If you specify a DATADIR, all Sybase data is stored as normal files in that directory. The default sizes are 10 GB for data, 3 GB for logs, and 2 GB for the temporary storage of Sybase. During the installation process, ownership of the directory changes to the Sybase user. A fresh install requires an empty "DATADIR" directory. Upgrade option upgrades the installed Sybase version to Sybase in the same directory. System JRE should be version 1.6 or later while using Sybase version Using an external database, such as MySQL or Oracle, requires creating a schema and DB USER with the required privileges. Reconfigure Collection Manager to the database using the ~scmscm/scripts/dbconf.sh script option. Upgrading the Bundled Sybase Database Starting from SCE Release 4.0.0, you can upgrade the bundled Sybase database. Sybase is supported only on Red Hat Enterprise Linux 5.x, CentOS 5.x, and Solaris 10. Actions the installsyb.sh Script Performs During an Upgrade The installsyb.sh script upgrades the Sybase database. The script performs the following actions: Checks the existence of user Sybase and group Sybase. If the User exists, and the Sybase is running as the existing user, the script stops the Sybase server. Runs the Sybase installer for the platform. Starts Sybase. Prerequisites Log in as the root user and make the distribution kit contents available on your system or local network. To upgrade the existing bundled Sybase database to version : Step 1 Change directory to Sybase in the distribution kit root. Before starting the upgrade, make sure 1 GB of free memory is available for the Sybase. Step 2 Run the installsyb.sh script. installsyb.sh --upgrade 3-11

52 Installing the Collection Manager Software Chapter 3 Installing Collection Manager The upgrade option upgrades the installed Sybase version to Sybase version in the same directory. System JRE should be version 1.6 or later while using Sybase version Installing the Collection Manager Software This section describes how to install Collection Manager. Information About the install-cm.sh Script To install Collection Manager server, use the install-cm.sh script. install-cm.sh Options The usage message for the install-cm.sh script is: Usage: install-cm.sh [-h] (-d CMDIR -o) Options: -d CMDIR select directory for ~scmscm (must not exist and must be on 8 GB free partition) -o upgrade the existing installation while preserving the current configuration (can't be used with -d) -h print this help and exit Description of the options: -d CMDIR -o Used to designate the directory of the newly created scmscm user's home. Should be the name of a non-existing directory, whose parent resides on a partition where at least 8 GB is free. As an alternate to this option, you can specify -o : Use this option when you wish to upgrade the existing installation while preserving the current configuration. (can't be used with -d) Actions install-cm.sh Performs The install-cm.sh script performs the following actions: If needed, creates an scmscm user and a scmscm group. Optionally, creates the home for this user Populates the home of scmscm with Collection Manager files and scripts Installs the extra component private JRE in the ~scmscm/cm/lib library. Creates boot script symbolic links for the Sybase and scmscm users in /etc/init.d and /etc/rcx.d All Collection Manager scripts run in the Bash shell environment. No other shells are supported. 3-12

53 Chapter 3 Installing Collection Manager Installing the Collection Manager Software Step 1 Step 2 Step 3 Step 4 Step 5 Change the directory to install-scripts under the distribution kit root. Run the install-cm.sh script. After running the script, a user-driven configuration manager presents options for the basic configuration of Collection Manager. Choose whether to enable the verbose mode. If enabled, verbose mode displays a detailed installation log on the console. The default value is no. Do you want to enable verbose mode (for detailed installation) (yes/no)[no]: You can choose to set up the Collection Manager database at the time of installation or configure it at a later time. The default value is yes. Would you like to configure the database? (yes/no)[yes]: If you choose to configure Collection Manager database: a. Select the number corresponding to the relational database management system of the connecting database: Enter the DB type: 1 - Oracle 2 - MySQL 3 - Sybase 4 - exit Enter your choice: b. Enter the following server information or press enter to leave at the default shown. Enter DB server host (default localhost): Enter DB server listening port (default port no) : Enter DB server instance id (default schema) : Enter DB schema user name (default user_id) : Enter DB schema user password (default password): Do you want to test the DB connection? (yes/no): Please answer using "yes" or "no" If you chose yes, the following text appears: Testing the DB connection... PASS:db is up DB connection succeeded. Updating the time zone... NOTE: The user name created for SCMS-CM is scmscm. You may want to set a password for this account now. Step 6 Step 7 Step 8 Restart Collection Manager for the database configuration to take effect. To start Collection Manager, choose yes: Would you like to start CM now? (yes/no)[yes]: The most basic installation of Collection Manager software is done. (Optional) Set a password. Run the following command to set the password for the scmscm user: passwd scmscm Be sure to record the password that you choose. 3-13

54 Installing the Collection Manager Software Chapter 3 Installing Collection Manager Step 9 (Optional) Increase the amount of memory allocated to the TA adapter. An application that uses the TA adapter may require an increase in the amount of allocated memory. The need to increase memory depends on the number of subscribers Collection Manager manages. To increase the memory allocation: a. Open the file ~scmscm/cm/config/cm.conf. b. Locate the setting containing TAAdapter in the [adapter_mem] section. c. Change the default value (512 MB) to a larger value. For example, to allocate 1024 MB of memory, set the value to -Xmx1024M. d. Save and close the file. Step 10 For information about calculating the TA Adapter memory, see the TA Adapter Memory Requirements section on page 2-9. (Optional) Increase the amount of memory allocated to the RAG adapter. An application that uses the RAG adapter can require an increase in the amount of allocated memory. The need to increase memory depends on the number of subscribers Collection Manager manages and on the RAG Adapter configuration. To change the setting: a. Open the file ~scmscm/cm/config/cm.conf. b. Locate the setting containing RAGAdapter in the [adapter_mem] section. c. Change the default value (512 MB) to a larger value. For example, to allocate 1024 MB of memory, set the value to -Xmx1024M. d. Save and close the file. To use an external database, configure a dbpack to connect Collection Manager to the database. See Chapter 5, Managing Databases and the Comma-Separated Value Repository for instructions. The app_conf_dir values of the JDBC (jdbcadapter.conf) and TA (taadapter.conf) adapter configuration files point to the most recent directory of the desired applications by default. Step 11 On Red Hat Enterprise Linux, make sure that scmscm user has read permission to generate the Cisco Service Control Collection Manager support files along with the /var/log/messages* files. (Optional) Increase the amount of memory allocated to the Custom adapter. The default value is 512 MB. Applications that use Custom adapter may require more memory allocated, depending on the number of subscribers the Cisco Service Control Collection Manager manages and on the Custom Adapter configuration. To change the allocated memory: a. Open the file ~scmscm/cm/config/cm.conf. b. Locate the setting containing CustomAdapter in the [adapter_mem] section. 3-14

55 Chapter 3 Installing Collection Manager Installing the Collection Manager Software Step 12 c. Change the default value to a higher value. For example, to allocate 1024 MB of memory, set the value to -Xmx1024M. d. Save and close the file. (Optional) Install and activate the periodic delete procedures for the database tables. (For information about configuring the behavior of periodic delete, see the Managing the Periodic Deletion of Old Records section on page 5-3.) If reports are sent to the database, and you do not install and activate the periodic delete procedures, the second disk might overflow. a. Install the periodic delete procedures. Log in as the scmscm user, start Collection Manager. Wait 1-2 minutes for the creation of the database tables. Run the script: ~scmscm/db_maint/create_periodic_del_procs.sh b. Activate the automatic invocation of the periodic delete procedures by running the following command: ~scmscm/scripts/dbperiodic.sh --load Step 13 Start Collection Manager by running the ~scmscm/cm/bin/cm start. The script performs the following actions: Starts all the adapters enabled in Collection Manager. Populates the IMEI_DEVICETYPE table with default values. This occurs when the table is not created and no records exist in the table. Populates the OSFP_TYPES table with default values upon starting or restarting Collection Manager if the OSFP detection is enabled. 3-15

56 Configuring the External MySQL Server Chapter 3 Installing Collection Manager Configuring the External MySQL Server You can configure Collection Manager to operate with an external server operating the MySQL Server 4.1 software or the MySQL Server 5.0 software. To do this, log in to the MySQL Server and manually create a schema and a user with necessary privileges before configuring Collection Manager. Log in to the server using root privileges and execute the following MySQL commands to create pqb_admin with root privileges: create database apricot; create user 'pqb_admin' identified by 'pqb_admin'; grant all privileges on *.* to 'pqb_admin'@'localhost' identified by 'pqb_admin' with grant option; grant all privileges on *.* to 'pqb_admin'@'<cm_server_ip>' identified by 'pqb_admin' with grant option; grant all privileges on *.* to 'pqb_admin'@'%' identified by 'pqb_admin' with grant option; This set of privileges will grant root privilege to the pqb_admin user. Redefine user privileges to restrict user access. 3-16

57 Chapter 3 Installing Collection Manager Uninstalling the Sybase Database and Collection Manager Software Uninstalling the Sybase Database and Collection Manager Software The following sections describe how to uninstall the Sybase database and Collection Manager: Uninstalling Sybase, page 3-17 Uninstalling Collection Manager Software, page 3-17 Uninstalling Sybase To uninstall the Sybase database, perform the following steps: Step 1 Step 2 Step 3 Log in to the Collection Manager as the root user. Uninstall Sybase. Change the directory to install-scripts under the distribution kit root directory, and enter:./uninstall.sh --sybase [datadir] [datadir] value is optional. If the [datadir] option is entered, the Sybase data files are removed. Edit the /etc/system file for Solaris or the /etc/sysctl.conf file for Red Hat Linux and remove the Sybase shmem setting. Uninstalling Collection Manager Software To uninstall Collection Manager software, perform the following steps: Step 1 Step 2 Log in as the root user. Uninstall Collection Manager software. Change the directory to install-scripts under the distribution kit root directory, and enter:./uninstall.sh --cm 3-17

58 Upgrading Collection Manager to Version 4.0.x Chapter 3 Installing Collection Manager Upgrading Collection Manager to Version 4.0.x You can upgrade to Cisco Service Control Collection Manager Release 4.0.x from Cisco Service Control Collection Manager Release and later. To upgrade to Release 4.0.x from a version earlier than 3.7.2, you must first upgrade to version To upgrade the CM:RDR database with 4.0.x compatibility, perform the following steps: Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Download the Collection Manager software, as described in the Getting Collection Manager Software, page 3-8. Change directory to install-scripts under the distribution kit root. Log in as the scmscm user, stop the Collection Manager server: $ ~scmscm/cm/bin/cm stop Log in as root user, run the install-cm.sh script: #./install-cm.sh -o After running the script, a user-driven configuration manager presents options to configure the Cisco Service Control Collection Manager. Choose whether to enable the verbose mode. If enabled, verbose mode displays a detailed installation log on the console. The default value is no. Do you want to enable verbose mode (for detailed installation) (yes/no)[no]: no Choose whether you want to upgrade the Cisco Service Control Collection Manager tables to the latest schema. The default value is yes. Do you want to upgrade the CM Tables to latest Schema? (It will take time if the table have huge data)(yes/no)[yes]: yes Step 7 You must upgrade the tables to the latest schema before starting the Collection Manager process. If the no option is selected, you must run the ~scmscm/scripts/ upgraderdrtables.sh script as a scmscm user manually. After the upgrade, when the Collection Manager comes up for the first time, the database tables that are new in Version 4.0.x are created automatically. (Optional) Choose whether to start the Cisco Service Control Collection Manager. The default value is yes. If you do not want to start the Cisco Service Control Collection Manager, choose no. Would you like to start CM now? (yes/no)[yes]: yes If you enter no, manually start the Cisco Service Control Collection Manager using the ~scmscm/cm/bin/cm start command. 3-18

59 CHAPTER 4 Managing Collection Manager This chapter explains how to use utility scripts to view and update Collection Manager parameters. Any machine connected to Collection Manager through, for example, Telnet or Secure Shell (SSH), can use utility scripts to monitor and manage Collection Manager. The utility scripts are located in the installation directory of Collection Manager. For information on managing the database and the CSV repository, see Chapter 5, Managing Databases and the Comma-Separated Value Repository. This chapter contains the following sections: Using Utility Scripts, page 4-2 Collection Manager Support Information, page 4-2 Configuring Collection Manager, page 4-3 Configuring the Categorizer, page 4-7 Monitoring System Health, page 4-8 Managing Users, page 4-10 Managing Virtual Links, page 4-11 Managing IMEI or MEID TAC with Device Types, page 4-12 Managing the Peak Hour Configuration, page 4-14 Monitoring Collection Manager, page

60 Using Utility Scripts Chapter 4 Managing Collection Manager Using Utility Scripts General instructions for using utility scripts include: To invoke any script, log in as the scmscm user, except when otherwise specified. If you attempt to run these scripts as the root user, the script generates an error. To display a description of the script, with an explanation of all the flags and parameters, invoke the script with the help flag. Specify the help flag differently depending on the script. To manage Collection Manager, specify -help. To manage the database, specify -h. The following example shows how to display a description of the dbperiodic.sh script: >~scmscm/scripts/dbperiodic.sh --help Usage: ~scmscm/scripts/dbperiodic.sh --load load configuration from /export/home/scmscm/db_maint/dbperiodic.conf ~scmscm/scripts/dbperiodic.sh --loadfile=file load configuration from FILE ~scmscm/scripts/dbperiodic.sh --dump print the current configuration in INI format to standard output ~scmscm/scripts/dbperiodic.sh --help print this help message Collection Manager Support Information If you are experiencing difficulties with Collection Manager installation, you might need to provide the Cisco Technical Assistance Center (TAC) with information about the current system setup. Collection Manager can create the required support information or files for TAC by running one of the following scripts: ~scmscm/unsupported/getinfo/get_cm_info.sh This script generates the Collection Manager support information. ~scmscm/unsupported/getinfo/get_support_files.sh This script creates a support file. The output is a zip file with the investigation supporting files. The following example shows the output that results from running the script: bash-2.05$ ~scmscm/unsupported/getinfo/get_support_files.sh Gathering support files into Sun_14_Oct_2007_ _PM_JST.zip, please wait...done. The generated support file contains the output of ~scmscm/unsupported/getinfo/get_cm_info.sh 4-2

61 Chapter 4 Managing Collection Manager Configuring Collection Manager Configuring Collection Manager You can use both scripts and files to configure Collection Manager. The following sections provide details on these options: Using Scripts to Configure the Collection Manager, page 4-3 Using Files to Configure the Collection Manager, page 4-6 Using Scripts to Configure the Collection Manager Use utility scripts to: Specify which servers to activate at startup. Start or stop the database. Start or stop an adapter. Drop an SCE connection. Use the following scripts to configure Collection Manager: ~scmscm/setup/on-boot.sh ~scmscm/scripts/adapterconf.sh ~scmscm/scripts/sceconf.sh ~scmscm/scripts/dbconf.sh For information about the scripts used for managing the database and the CSV repository, see Chapter 5, Managing Databases and the Comma-Separated Value Repository. The below sections detail the scripts used to configure the Collection Manager. Activating Servers, page 4-3 Controlling Adapters, page 4-4 Enabling an Adapter, page 4-5 Dropping a Cisco SCE Connection, page 4-5 Configuring a Database, page 4-5 Configuring the Timezone Offset, page 4-6 Activating Servers To specify the servers (Collection Manager or Sybase) that are activated at startup, use the on-boot.sh script: ~scmscm/setup/on-boot.sh --cm=flag --sybase=flag The table Table 4-1defines the on-boot.sh script parameters: 4-3

62 Controlling Adapters Chapter 4 Managing Collection Manager Table 4-1 on-boot.sh Parameters --cm={ on off } --sybase={ on off } Activate or do not activate Collection Manager at startup. Activate or do not activate the Sybase server at startup. The following example shows how to set Collection Manager and Sybase servers to run at startup. (This setting is the default setting of the script.) >~scmscm/setup/on-boot.sh -cm=on -sybase=on To view the current startup status of each component, run the on-boot.sh script with no parameters. Server specification take effect when the system restarts. To restart Collection Manager, run the following script as the scmscm user: ~scmscm/cm/bin/cm restart Controlling Adapters To shut down or activate a configured adapter, use the adapterconf.sh script. You can use the adapterconf.sh script to list Collection Manager adapters that are currently running: ~scmscm/scripts/adapterconf.sh --op=action [ --adapter=adapter name ] The table Table 4-2 shows the adapterconf.sh script options: Table 4-2 adapterconf.sh Options --op=start --op=stop --op=list adapter=adapter name --help Activate the adapter specified by the adapter parameter. Shut down the adapter specified by the adapter parameter. List Collection Manager adapters that are currently running. Identify the adapter you want to control. Use only with the start and stop options. Display these options. To shut down an adapter, as the scmscm user, run the following script: ~scmscm/scripts/adapterconf.sh --op=stop -adapter=adapter name To activate an adapter, as the scmscm user, run the following script: ~scmscm/scripts/adapterconf.sh --op=start -adapter=adapter name 4-4

63 Chapter 4 Managing Collection Manager Enabling an Adapter Enabling an Adapter To configure an adapter to turn on when Collection Manager starts, remove the remark character at the start of the appropriate line in the cm.conf file. The following example shows how to define the RAG adapter to turn on when Collection Manager starts. adapter.4 = com.cisco.scmscm.adapters.rag.ragadapter The following example shows how to configure the CSV adapter to remain off when Collection Manager starts: #adapter.2 = com.cisco.scmscm.adapters.csvadapter The value of the adapter.<number> must match the adapter_id parameter value configured in the queue.conf file for the corresponding adapter. Dropping a Cisco SCE Connection To drop a connection to a particular Cisco SCE, run the sceconf.sh script: ~scmscm/scripts/sceconf.sh --op=drop --ip=ip address Run this script only if the HTTP adaptor of Collection Manager is running. Also use this script to display information about the Cisco SCE connection. (See the Checking the SCE Connection section on page 4-16) Table 4-3 shows the sceconf.sh script options: Table 4-3 sceconf.sh Options Adapter=IP address --help Drop the connection at the specified IP address. Display the seconf.sh options. To drop a connection, as the scmscm user, run the following command: ~scmscm/scripts/sceconf.sh --op=drop --ip=ip address Configuring a Database To configure a database, run the dbconf.sh script: ~scmscm/scripts/dbconf.sh The script prompts you to choose the database type and the corresponding database configuration parameters. The following example shows how to run the dbconf.sh script to configure Collection Manager to work with an external Oracle database: $ ~scmscm/scripts/dbconf.sh Enter the DB type: 1 - Oracle 2 - MySQL 3 - Sybase 4 - exit 4-5

64 Configuring the Timezone Offset Chapter 4 Managing Collection Manager Enter your choice: 1 Enter Oracle server host (current is localhost) : Enter Oracle server listening port (current is 1521) : Enter Oracle server instance id (current is apricot) : Enter CM schema user name (current is pqb_admin) : Enter CM schema user password (current is pqb_admin) : Do you want to test the DB connection? (yes/no): yes Testing the DB connection... PASS:db is up DB connection succeeded. Updating the time zone... Restart Collection Manager for the configuration to take effect. Configuring the Timezone Offset You can configure the timezone offset manually or automatically. To configure the timezone offset (from GMT) manually, issue the following command: ~scmscm/cm/bin/jselect-sce-tz.sh --offset=offset-in-minutes Enter the offset in minutes. For example, if the SCE is located in GMT+2, issue the following command: ~scmscm/cm/bin/jselect-sce-tz.sh --offset=120 For example, if the SCE is located in GMT-10, issue the following command: ~scmscm/cm/bin/jselect-sce-tz.sh --offset=-600 To configure the timezone offset (from GMT) automatically, issue the following command: ~scmscm/cm/bin/updatetimezone.sh The updatetimezone.sh script calculates the offset in minutes and updates the CONF_SE_TZ_OFFSET table. Run the updatetimezone.sh script every time the timezone of the Cisco SCE is updated. For example, when updating the timezone to accommodate daylight savings time, remember to run the updatetimezone.sh script. The SCA Reporter does not use the correct time intervals unless the offset stored in Collection Manager is consistent with the timezone of the SCE. Using Files to Configure the Collection Manager Use the following files to configure Collection Manager: cm.conf General configuration of Collection Manager, including the adapters to turn on when Collection Manager starts. See the Enabling an Adapter section on page 4-5. queue.conf Configuration of the adapter queues, including the RDR tags that are associated with a specific adapter. See the Configuring the Categorizer section on page

65 Chapter 4 Managing Collection Manager Configuring the Categorizer Configuring the Categorizer The Categorizer classifies each RDR according to its RDR tag. You can route an RDR to a specific adapter by adding its RDR tag to the tags parameter (a comma-separated list of RDR tags) of the adapter. This configuration is contained in the queue.conf file. The following example shows how to configure the RDR tags and to be sent to the topper/aggregator (TA) adapter. # Topper/Aggregator Adapter [topper-hi] adapter_id=3 priority=3 warning_size=40000 maximum_size=50000 tags= , The value of the adapter_id parameter must match the adapter.<number> defined in the cm.conf file for the corresponding adapter. 4-7

66 Monitoring System Health Chapter 4 Managing Collection Manager Monitoring System Health Collection Manager contains a small, expandable framework that monitors the system and issues alerts for predefined, potentially problematic conditions. Use the following scripts to monitor Collection Manager: ~scmscm/setup/monitor/setup-monitor.sh ~scmscm/setup/monitor/monitor.sh Installing the Periodic Checker To insert (or remove) an entry for monitor.sh, the periodic checker script, in the cron (periodic scheduler) subsystem, run the setup-monitor.sh script: ~scmscm/setup/monitor/setup-monitor.sh -a flag [ -I flag ] Table 4-4 shows the setup-monitor.sh script options. Table 4-4 setup-monitor.sh Options -a { install uninstall } Insert or remove an entry for monitor.sh in the cron. -i { 30m 1h 12h 24h } Run monitor.sh every 30 minutes, 1 hour, 12 hours, or 24 hours. The following example shows how to install monitor.sh so that it runs every 30 minutes: $./setup-monitor.sh -a install -i 30m The following example shows how to uninstall monitor.sh: $./setup-monitor.sh -a uninstall Periodic Checker Script Periodic Checker Script The following sections provide information on the periodic checker script: Periodic Checker Script, page 4-8 Tests, page 4-9 The periodic checker script, monitor.sh, calls a series of subscripts that monitor different aspects of a running system: ~scmscm/setup/monitor/monitor.sh { -a TEST NAME } [ -v ] [ -d ] The script is not intended to be run from the command line, although you can do so. The script sends test results to the syslog subsystem and logs the results in the /var/log/messages file. Table 4-5 shows the smonitor.sh script options. 4-8

67 Chapter 4 Managing Collection Manager Periodic Checker Script Table 4-5 monitor.sh Options -a Run all tests. TEST NAME The names of one or more tests. A test name is the test filename, without the leading digits and trailing.sh. -v Output results in verbose mode. (Log successful tests.) -d Print results to window. (By default, results are sent to syslog.) Any test returns a result in the following format: -STATUS: Message STATUS PASS or FAIL Message A short informative status message For example, FAIL: db "apricot" has only 1523 free blocks The following example shows how to run all available tests and print system output to the window: $./monitor.sh -d -a Test: 01free_db.sh. Status: PASS. Message: db apricot has 1532 free blocks Test: 02cm_is_up.sh. Status: FAIL. Message: cm process is not running The following example shows how to run one test to check the installed database has sufficient free space: $./monitor.sh -d free_db Test: 01free_db.sh. Status: PASS. Message: db apricot has 1532 free blocks Tests You can run the following tests using monitor.sh: db_up Checks whether the Collection Manager database is running. cm_up Checks whether the Collection Manager application is running. free_db Checks whether the bundled Sybase database has at least 10 percent free space. free_log Checks whether the bundled Sybase database transaction log has at least 70 percent free space. cm_persistent_buffers Checks whether the persistent buffer of each Collection Manager adapter contains less than 500 files. The scripts for all these tests are located in the ~/setup/monitor/tests directory. When calling a test called test_name, the script expects to find a file called NNtest_name.sh, where NN is a number that denotes script priority. For example, the test free_db is mapped to the file 01free_db.sh. 4-9

68 Managing Users Chapter 4 Managing Collection Manager Managing Users Collection Manager uses the p3rpc utility to manage users for authenticated RPC calls. The command format is: p3rpc OPERATION [OPTIONS] Table Table 4-6 lists the p3rpc operations and options. Table 4-6 p3rpc Operations Operation --set-user --username=username --password=password --validate-password --username=username --password=password --delete-user --username=username --show-users Description Adds and updates the username and password. Validates the username and password. Deletes a user configuration. Displays all the configured users. Examples The following example shows how to add and update a username and password: bash-2.05$ p3rpc --set-user --username=lulu --password=lili Command terminated successfully bash-2.05$ The following example shows how to validate a username and password. In this example, the user is successfully validated: bash-2.05$ p3rpc --validate-password --username=lala --password=lala Local machine: user lala was authenticated successfully : (auth level root) Command terminated successfully bash-2.05$ The following example shows how to validate a username and password. In this example, the user validation fails: bash-2.05$ p3rpc --validate-password --username=lala --password=lulu Error - Failed to authenticate user lala bash-2.05$ The following example shows how to delete a user configuration: bash-2.05$ p3rpc --delete-user --username=lulu Command terminated successfully bash-2.05$ The following example shows how to display all of the configured users. In this example, only one configured user, clu, exists: bash-2.05$ p3rpc --show-users clu Command terminated successfully bash-2.05$ 4-10

69 Chapter 4 Managing Collection Manager Managing Virtual Links Managing Virtual Links A script is included in Collection Manager distribution to enable you to manage virtual link names and indexes that are configured for a specific SCE: To show or set virtual links, run the update_vlinks.sh script: ~scmscm/cm/bin/update_vlinks.sh --sce=sce IP address [ --file=file --show ] Table 4-7 shows the update_vlinks.sh script options. Table 4-7 update_vlinks.sh Options --sce=sce IP --file=file --sce=sce IP --show --help Update the VLINK_INI table with the data in the supplied CSV formatted file for the specified SCE. Query the VLINK_INI table for entries for the specified SCE. Display the above options. To set the virtual link details, as the scmscm user, run the following command: ~scmscm/cm/bin/update_vlinks.sh --sce=sce IP address --file=file The CSV file format is link ID (positive integer), link direction (0=upstream, 1=downstream), name (string). The script performs following validations on the file: The file exists. There are no duplicate virtual links IDs for each direction. The virtual link ID is a positive value from 0 to The direction is either 0 (upstream) or 1 (downstream). No duplicate virtual link names or empty names exist for each direction. Virtual link names can contain up to 256 characters. All printable characters with an ASCII code from 32 through 126 (inclusive) can be used, except for 34 ("), 39 ('), and 96 (`). After the file is successfully validated, the script performs the following actions: Step 1 Step 2 Step 3 Step 4 The input CSVfile must contain three data fields (VLINK_ID, VLINK_DIRECTION, VLINK_NAME) or six data fields (VLINK_ID, VLINK_DIRECTION, VLINK_NAME, CHANNEL_ID, CHANNEL_NAME, CMTS_NAME). If the CSV file contains three data fields, the VLINK_INI table automatically fills the data for the rest of the columns. All entries in the table that match the SCE_IP_Address and CHANNEL_ID, VLINK_DIRECTION, and CHANNEL_NAME are deleted from the table. The script adds two entries to the VLINK_INI table in the following format: Timestamp, sce ip, 0, 0, "Default Virtual Link Up,0, 0, "Default Virtual Link Up" Timestamp, sce ip, 0, 1, "Default Virtual Link Down, 0, 1, "Default Virtual Link Down" 4-11

70 Managing IMEI or MEID TAC with Device Types Chapter 4 Managing Collection Manager Step 5 Step 6 The script parses the CSV file and each line in the CSV file is entered as a record entry in the VLINK_INI table. If the CSV file is empty, the script inserts default records into the VLINK_INI table. Managing IMEI or MEID TAC with Device Types Collection Manager supports either GSM or CDMA mobile-based reports. GSM reports are enabled by default. Once Collection Manager is installed, the CSV file ~scmscm/cm/config/imeitac_device_map.csv populates the IMEI_DEVICETYPE table with default values. The MEID_DEVICETYPE table stores the CDMA Device Type mapping information in the database. You manually execute the ~scmscm/scripts/update_devicetype.sh script to update the table. The default input mapping file is ~scmscm/cm/config/meidtac_device_map.csv: ~scmscm/scripts/update_devicetype.sh Table 4-8 shows the update_devicetype.sh Options Table 4-8 update_devicetype.sh Options update_devicetype.sh --imei --r update_devicetype.sh --meid --r update_devicetype.sh --imei --u update_devicetype.sh --meid --u update_devicetype.sh --h Removes all the existing records from the IMEI_DEVICETYPE Table. Removes all the existing records from the MEID_DEVICETYPE Table. Deletes all the existing records and then updates the IMEI_DEVICETYPE table with the data given in the default CSV file. Deletes all the existing records and then updates the MEID_DEVICETYPE table with the data given in the default CSV file. Prints the usage messsage. Update the IMEI_DEVICETYPE table To update the IMEI_DEVICETYPE table, add new TAC details to the CSV file ~scmscm/cm/config/imeitac_device_map.csv in following the format: Manufacturer <string>, Model <string>, TAC <number>. Next, run the /update_devicetype.sh script using the --u option. ~scmscm/scripts/update_devicetype.sh --imei --u This deletes all the existing records, then updates the IMEI_DEVICETYPE table with the data given in the default CSV file. The update_devicetype.sh script performs the following actions: The script parses the CSV file ~scmscm/cm/config/imeitac_device_map.csv, and each line in the CSV file is entered as a record entry in the IMEI_DEVICETYPE table. 4-12

71 Chapter 4 Managing Collection Manager Update the MEID_DEVICETYPE Table If the TAC number is a positive value, records are considered valid. Otherwise, the records are ignored. If the script finds duplicate TAC entries, the first TAC entry is considered valid while the other entries are ignored. If the CSV file is empty, the script deletes all the existing records in the IMEI_DEVICETYPE table. Update the MEID_DEVICETYPE Table To update the MEID_DEVICETYPE table: Step 1 Step 2 Add the new TAC details to the ~scmscm/cm/config/meidtac_device_map.csv file in the format Manufacturer <string>, TAC <number>. Run the update_devicetype.sh script using the --u. ~scmscm/scripts/update_devicetype.sh --meid --u This deletes all the existing records then updates the MEID_DEVICETYPE table with the data given in the default CSV file. The update_devicetype.sh script performs the following actions: The script parses the CSV file ~scmscm/cm/config/meidtac_device_map.csv and each line in the CSV file is entered as a record entry in the MEID_DEVICETYPE table. If the TAC number is a positive value, records are considered valid. Otherwise, the records are ignored. If the script finds duplicate TAC entries, the first TAC entry is considered valid while the other entries are ignored. If the CSV file is empty, the script deletes all the existing records in the MEID_DEVICETYPE table. Update the OSFP_TYPES Table To update the OSFP_TYPES table: Step 1 Step 2 Add the new Operating System details to the ~scmscm/cm/config/osfp_types_map.csv file in the format OS id <number>,os name <string>. Run the update_osfptypes.sh script using --update. ~scmscm/scripts/update_osfptypes.sh --update This deletes all the existing records then updates the OSFP_TYPES table with the data given in the default CSV file. The update_osfptypes.sh script performs the following actions: The script parses the CSV file ~scmscm/cm/config/osfp_types_map.csv and each line in the CSV file is entered as a record entry in the OSFT_TYPES table. If the CSV file is empty, the script deletes all the existing records in the OSFP_TYPES table. 4-13

72 Managing the Peak Hour Configuration Chapter 4 Managing Collection Manager Managing the Peak Hour Configuration You need to execute the ~scmscm/scripts/updatepeakconfig.sh script before generating the peak hour reports. This script populates the peak_hour_config table with the default peak hour details. By default, the peak_hour_config table is not created. The updatepeakconfig.sh script then prompts you to enter the peak hour range for any days you want to change from the default. If the value is not entered for the corresponding day, the current value shown in the prompt is considered the default value. Execute the ~scmscm/scripts/updatepeakconfig.sh script once with default values before generating any peak hour reports. The following output is an example of a Peak Hours configuration: -bash-2.05b$ ~scmscm/scripts/updatepeakconfig.sh Enter the Peak Hour Range for Monday (current is 18-21) : Enter the Peak Hour Range for Tuesday (current is 18-21) :15-16 Enter the Peak Hour Range for Wednesday (current is 18-21) :15 Enter the Peak Hour Range for Thursday (current is 18-21) : Enter the Peak Hour Range for Friday (current is 18-21) : Enter the Peak Hour Range for Saturday (current is 16-19) : Enter the Peak Hour Range for Sunday (current is 16-19) : Peak Configurations successfully updated!!!!!restart CM for the new config to take effect!! 4-14

73 Chapter 4 Managing Collection Manager Monitoring Collection Manager Monitoring Collection Manager You can use scripts to monitor system statistics that are relevant to Collection Manager, such as: Percentage of free space in the database Rate of RDRs entering Collection Manager SCE platform connection data Viewing database insertion rate statistics per table Viewing version information Use the following scripts to monitor Collection Manager: ~scmscm/scripts/dbfree.sh ~scmscm/scripts/rdr-rate.sh ~scmscm/scripts/sceconf.sh ~scmscm/setup/alive.sh ~scmscm/cm/bin/p3stats ~scmscm/cm/bin/cm version ~scmscm/cm/bin/cm dbversion Use the following scripts to configure Collection Manager (see the Configuring Collection Manager section on page 4-3). You can also run these scripts to display the relevant configuration: ~scmscm/setup/on-boot.sh ~scmscm/scripts/adapterconf.sh Checking the Database Capacity To display the percentage of free space in the database report tables and the associated transaction log, run the dbfree.sh script: ~scmscm/scripts/dbfree.sh Use the script only with a bundled Sybase and MySql databases. Step 1 As the scmscm user, run the dbfree.sh script. Checking the RDR Rate To display the momentary total rate of reports entering Collection Manager, run the rdr-rate.sh script rdr-rate.sh The output is a single floating-point number that represents the total rate per second of incoming RDRs (from all sources), which entered Collection Manager in the past 5 seconds. Use this script only if the HTTP adaptor of Collection Manager is running. 4-15

74 Checking the SCE Connection Chapter 4 Managing Collection Manager Step 1 As the scmscm user, run the rdr-rate.sh script. Checking the SCE Connection To display information about the SCE connections, run the sceconf.sh script: ~scmscm/scripts/sceconf.sh --op=list Use this script only if the HTTP Adaptor of Collection Manager is running. Also use the script to drop a connection from a particular SCE. See the Dropping a Cisco SCE Connection section on page 4-5. Step 1 As the scmscm user, run the sceconf.sh script. ~scmscm/scripts/sceconf.sh --op=list Example The following example shows SCE connection output: >~scmscm/scripts/sceconf.sh --op=list IP Rate Peak Verifying Server Operation To verify that the server is functioning correctly, run the alive.sh script: ~scmscm/setup/alive.sh The script verifies that the following components are operational: Collection Manager Database (in the bundled database case) Report tables (in the bundled database case) If any component is down, the script issues an error message. Step 1 As the scmscm user, run the alive.sh script It takes time for the components to initialize after a startup. After a restart, wait 5 minutes before running this script. 4-16

75 Chapter 4 Managing Collection Manager Viewing Database Statistics Viewing Database Statistics To view the statistics of Collection Manager database, run the p3stats command line utility (CLU). ~scmscm/cm/bin/p3stats [options] Table 4-9 lists the p3stats options. Table 4-9 p3stats Options Option --show-stats --reset-stats Description Displays statistics information in the format <DB RDR tag> <RDR rate> <peak RDR rate> Resets the statistics counters. Collection Manager must be running for this CLU to operate. The following example shows how to display the statistics of Collection Manager database: $ ~scmscm/cm/bin/p3stats --show-stats LUR { rate=0, peak=0 } MALUR { rate=0, peak=0 } TR { rate=0, peak=0 } VLUR { rate=0, peak=0 } NUR { rate=0, peak=0 } SUR { rate=0, peak=0 } MEDIA { rate=0, peak=0 } PUR { rate=0, peak=0 } Command terminated successfully $ Viewing Version Information Collection Manager includes two scripts to display the current version of Collection Manager and of the database: cm version cm dbversion The following example shows how to view the current Collection Manager version: $ ~scmscm/cm/bin/cm version CM CD Version Build 45 The following example shows how to view the current database version: $ ~scmscm/cm/bin/cm dbversion CM: Database Sybase ase

76 Viewing Version Information Chapter 4 Managing Collection Manager 4-18

77 CHAPTER 5 Managing Databases and the Comma-Separated Value Repository This chapter explains how to use utility scripts to manage Collection Manager database and the comma separated value (CSV) repository. This chapter contains the following sections: Common Database Management Tasks, page 5-2 Managing the Bundled Sybase Database, page 5-8 Managing the External MySQL Database, page 5-10 Partitioning the MySQL Database, page 5-10 Managing the Oracle Database, page 5-13 Aggregating Database Data, page 5-14 Managing the CSV Repository, page

78 Common Database Management Tasks Chapter 5 Managing Databases and the Comma-Separated Value Repository Common Database Management Tasks The following database management tasks apply to all the supported databases: Listing Database Tables, page 5-2 Managing the Periodic Deletion of Old Records, page 5-3 Deleting a Table, page 5-6 Deleting Old Records, page 5-6 Unicode Support for Sybase Database, page 5-7 The Collection Manager adds a timestamp to every raw data record (RDR) that it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables. Use the following scripts to configure and maintain the database: ~scmscm/scripts/dbtables.sh ~scmscm/scripts/dbperiodic.sh ~scmscm/db_maint/create_periodic_del_procs.sh ~scmscm/scripts/droptable.sh ~scmscm/scripts/prunetable.sh ~scmscm/scripts/sybutfsupport.sh ~scmscm/scripts/upgradedb.sh ~scmscm/scripts/upgraderdrtables.sh ~scmscm/scripts/partitions.sh ~scmscm/scripts/createaggevents.sh Listing Database Tables To display a list of all the tables in the database, run the dbtables.sh script: ~scmscm/scripts/dbtables.sh Where applicable, the script displays the number of lines in the table and the earliest and latest timestamps. For more information about using the Cisco Service Control Application Reporter to display the actual content of the tables, see the Cisco Service Control Application Reporter User Guide. Table 5-1 lists the dbtables.sh script options. Table 5-1 dbtables.sh Options Option Description -l Lists only the existing table names (without statistics). -c Lists the existing table names with their record counts. -a Includes the nonreport tables in the listing. -f Enables fast-line counting, using the client rather than the JDBC. 5-2

79 Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the Periodic Deletion of Old Records Table 5-1 dbtables.sh Options (continued) Option Description -t { sec_num } Displays the maximum waiting time, in seconds, for the response. The default is no timeout. -h Prints this help message and exits. -s table_nm Displays the statistics of a given table. The following is sample output of the dbtables.sh: >~scmscm/scripts/dbtables.sh -c Executing query... name num_lines min_time max_time RPT_SUR :46: :00: RPT_LUR :25: :06:05.45 RPT_VLUR :12: :18: RPT_PUR :45: :44: RPT_TR :12: :18: RPT_MALUR :46: :32:18.42 RPT_MEDIA :14: :16: RPT_TOPS_PERIOD :00: :00:00.16 RPT_TOPS_PERIOD :00: :00:00.1 Managing the Periodic Deletion of Old Records To manage the periodic deletion of old records, perform the following steps: 1. If the periodic delete procedures have not already been installed during the Collection Manager installation, install them: Log in as the scmscm user, start Collection Manager, wait for 1 to 2 minutes for the creation of the database tables, and then run the script: ~scmscm/db_maint/create_periodic_del_procs.sh 2. Edit the periodic delete configuration file. For more information, see the Configuring Periodic Delete section on page To apply the new configuration, run the dbperiodic.sh utility script. For more information, see the Applying the Periodic Delete Configuration File section on page 5-5. Caution If you enable both the aggregation and the periodic delete feature, ensure that you increase their respective configuration values. Periodic deletion of a table does not begin while a previous periodic deletion procedure is still running. This prevents excessive load on the database, which would otherwise degrade insertion performance in the adapters. When two or more tables are scheduled to be reduced at the same time, the tables are processed in the order in which they are listed in the periodic delete configuration file. For ease of configuration, you can schedule periodic deletion for all the tables consecutively in one schedule. All the periodic delete activities recorded in the system log file. For Solaris, the system log file is located in the /var/adm/messages directory; for Linux, the system log file is located in the /var/log/messages directory. 5-3

80 Managing the Periodic Deletion of Old Records Chapter 5 Managing Databases and the Comma-Separated Value Repository The following is an example of the system log file at the time when the periodic delete action occurs: Feb 17 05:00:07 app-redhat79 logger: /opt/scmscm/db_maint/periodic_del.sh for hourly all - running single del for RPT_LUR to 14 days Feb 17 05:00:09 app-redhat79 logger: Starting periodic delete for RPT_LUR keeping 14 days Feb 17 05:00:09 app-redhat79 logger: Deleted rows: 0 from the table RPT_LUR MySql server supports periodic delete from MySql 5.1 and earlier. Caution Be sure to disable the MySQL partitioning process in Collection Manager when you enable the periodic delete feature. Configuring Periodic Delete The following section describes how to configure and apply the Periodic Delete configuration file. Configuring Periodic Delete, page 5-4 Applying the Periodic Delete Configuration File, page 5-5 By default, the periodic delete configuration file (dbperiodic.conf) is located in the ~scmscm/db_maint/ directory. The file has a structure that is similar to an INI file, where each section describes a particular data reduction operation (for a specific set of tables), to be performed according to a specified schedule. The name of each section of the file is not used when the file is parsed. You can use any name you choose. Each section begins with the section name in brackets, and contains the parameters shown in Table 5-2. (Not all parameters are required in each section of the configuration file.) Separate the parameters and their values by an equal sign (=). Table 5-2 contains examples of periodic delete configuration files. Table 5-2 Parameters in a Periodic Delete Configuration File Parameter name Explanation Value Default Example active Specifies whether or not to use this section of the configuration file true/false true false tablenames daystokeep minute hour day month Names of the tables to which this section of the configurtion file applies Number of days to maintain records The time at which to perform the deletion in this section of the configuration file Names of tables separated by commas, or * for all tables * (all) RPT_SUR,RPT_LUR Positive integers , * 0 23, * 1 31, * 1 12, * 0 * (all) * (all) * (all) 0 0,4,8,12,16,20 1 1,3,5,7,9,11 5-4

81 Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the Periodic Deletion of Old Records Values for all the parameters except active and daystokeep can be either a single value, a list of values separated by commas, a range of values (two values separated by a dash), or an asterisk (*), which signifies all possible values. A range is not possible for tablenames. In the following example, all the fields are set to their default values: # This dbperiodic.conf file emulates the legacy style for periodic # deletion. All tables are processed every hour on the hour, and # records are kept for 14 days. [hourly all] active = true tablenames = * daystokeep = 14 minute = 0 hour = * In this example, all the tables are reduced at 4:30 A.M. This leaves 10 days of data in each table. In addition, the real-time tables are reduced every hour, leaving three days of data in each table. # This dbperiodic.conf file reduces all tables once a day and # real-time tables once an hour. [daily all] active = true tablenames = * daystokeep = 10 minute = 30 hour = 4 [hourly real-time] active = true tablenames = RPT_SUR,RPT_LUR,RPT_PUR daystokeep = 3 minute = 0 hour = * Applying the Periodic Delete Configuration File To load and apply a new periodic delete configuration file or to view the current file, run the dbperiodic.sh script: ~scmscm/scripts/dbperiodic.sh [ --dump ] [ --load --loadfile=path_to_dbperiodic.conf ] When the script is used to load a new configuration file, it parses the file, verifies its validity, and updates the scmscm user's crontab to reflect the changes. Table 5-3 lists the dbperiodic.sh options. Table 5-3 dbperiodic.sh Options Option --load --loadfile=path to periodic delete configuration file --dump --h Description Load the periodic delete configuration file from the /export/home/scmscm/db_maint/ dbperiodic.conf Load the periodic delete configuration file from the specified directory. Print the periodic delete configuration. Displays the above options 5-5

82 Deleting a Table Chapter 5 Managing Databases and the Comma-Separated Value Repository The following example shows how to print the current periodic delete configuration: ~scmscm/scripts/dbperiodic.sh --dump The script prints the loaded periodic delete configuration. If the current periodic delete configuration file is not yet loaded, the configuration can vary from the script output. The following example shows how to load the periodic delete configuration file from the ~scmscm/db_maint/dbperiodic.conf file: ~scmscm/scripts/dbperiodic.sh --load The following example shows how to load the periodic delete configuration file from a specified location: ~scmscm/scripts/dbperiodic.sh --loadfile=path_to_periodic_delete_configuration_file Deleting a Table To delete a table or all the current tables from the database, run the droptable.sh script: ~scmscm/scripts/droptable.sh [-f] tableparameter Table 5-4 lists the droptable.sh options. Table 5-4 droptable.sh Options Option table_name ALLTABLES Description Drop table_name from the database. Drop all the tables from the database. -f Drop the table by force (no questions asked or errors reported) -h Displays the above options. The following example shows how to drop a table named RPT_SUR from the database with no request for confirmation: ~scmscm/scripts/droptable.sh f RPT_SUR The following example shows how to drop all the tables from the database: ~scmscm/scripts/droptable.sh ALLTABLES Deleting Old Records To remove records from a database table based on the timestamps of the records, run the prunetable.sh script: ~scmscm/scripts/prunetable.sh [ -f ] num_days table_name 5-6

83 Chapter 5 Managing Databases and the Comma-Separated Value Repository Unicode Support for Sybase Database Table 5-5 lists the prunetable.sh options. Table 5-5 prunetable.sh Options Option num_days table_name Description The maximum age (in days) of records not to deleted. The table whose records will be deleted -f Drop by force (no questions asked or errors reported). -h Displays the above options. The following example shows how to delete all the records that are more than seven days old from a table named RPT_SUR. Because the f flag is not specified, there can be requests for confirmation and errors: >~scmscm/scripts/prunetable.sh 7 RPT_SUR Implement the Periodic delete procedure {Refer Section : Managing the CM:Periodic Deletion of Old Records } before invoking the prunetable.sh script Unicode Support for Sybase Database To add the UCS Transformation Format 8-bit (UTF-8) support for the Sybase database: Step 1 Step 2 Step 3 Login to Cisco Service Control Collection Manager as a root user. Run the ~scmscm/scripts/sybutfsupport.sh script. The script restarts the Sybase after enabling the UTF-8 support in the database. Restart the Cisco Service Control Collection Manager. The UTF-8 support is enabled on the Cisco Service Control Collection Manager. 5-7

84 Managing the Bundled Sybase Database Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the Bundled Sybase Database The task of managing the bundled database includes: Backing up and restoring a database Updating Sybase with a changed IP address Updating the Sybase database statistics Collection Manager adds a timestamp to each RDR it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables. Use the following scripts to maintain the bundled Sybase database: ~scmscm/scripts/sybback.sh ~scmscm/scripts/sybrestore.sh ~scmscm/db_maint/update_statistics.sh Backing Up the Database In Collection Manager Release and later, by default, the backup database files are stored in the Collection Manager's home directory under the Sybase directory. An example of the home directory is /home/scmscm/sybase. Consult the Sybase Backup Server product documentation for a complete Sybase backup. The bundled Collection Manager backup is only designed for backing up and restoring small amounts of data, for example, transferring small tables between servers. 5-8

85 Chapter 5 Managing Databases and the Comma-Separated Value Repository Updating Sybase with a Changed IP Address Updating Sybase with a Changed IP Address When you change the IP address, you must also update the Sybase server: Step 1 Step 2 Step 3 Step 4 Step 5 Change the IP address. Run the interface configuration script: /etc/sysconfig/network-scripts/ifcfg-eth0 Run the network restart script: /etc/rc.d/init.d/network restart Update the Sybase server. As the root user, issue the following command: ~scmscm/setup/syb_interfaces.sh Run the Sybase stop and start script as the root user: ~scmscm/setup/sybase stop/start Restart Collection Manager, run the following script as the scmscm user: ~scmscm/cm/bin/cm restart Updating Sybase Database Statistics To update the Sybase database statistics, run the update_statistics.sh script once a week at a time of low database load, for example, at night. ~scmscm/db_maint/update_statistics.sh 5-9

86 Managing the External MySQL Database Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the External MySQL Database The tasks of managing the MySQL database includes: Backing up and restoring a database Collection Manager adds a timestamp to each RDR it receives and stores in the database. This timestamp is used when various maintenance operations are performed on the database tables. Use the following scripts to maintain the MySQL database: ~scmscm/scripts/mysqlback.sh ~scmscm/cm/config/dbpacks/mysql/4.0.20/mysql.properties ~scmscm/scripts/mysqlrestore.sh Backing Up the MySQL Database To backup all the tables in the database, run the ~scmscm/scripts/mysqlback.sh script. By default all database backup files are placed in the backup_dir=/mysql/backup directory. To change the default backup location, update the property of the backup_dir field in the ~scmscm/cm/config/dbpacks/mysql/4.0.20/mysql.properties file. The destination location path is defined in the *.properties file. Modify the path if you want to change the default location. Restoring a MySQL Database To restore a database from the backup file created by the mysqlback.sh script, run the mysqlrestore.sh script: ~scmscm/scripts/mysqlrestore.sh Caution MySQL restore option overrites the existing table records. Partitioning the MySQL Database You can store records in an external MySQL database. When you configure MySQL as the database, partitioning is supported. Prerequisites to Configuring Partitioning, page 5-11 Configuring Partitioning, page

87 Chapter 5 Managing Databases and the Comma-Separated Value Repository Prerequisites to Configuring Partitioning Prerequisites to Configuring Partitioning The partitioning feature is supported only when the MySQL server meets the following conditions: MySQL Version or later open_files_limit value in my.cnf file should be set to 8000 for default partitioning configuration under the [mysqld] section thread_stack value in my.cnf file should be set to at least 256 k under the [mysqld] section Restart the MySQL server to enable the configuration changes to take effect. Configuring Partitioning Caution If a partition is created on the existing data, all the existing data will be deleted. Back up the existing data before running the partitioning script. The partitioning configuration file (partitions.conf) is located in the ~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions/ directory. The following is a sample of the configuration file: [partitions_config1] table_names = RPT_LUR,RPT_TOPS_PERIOD1,RPT_FUR,RPT_GUR no_of_partition = 30 period = 1 scheduler_time = 23:30 Each section begins with a unique section name starting with partitions_config in brackets and contains the parameters shown in Table 5-6. Separate the parameters and their values by an equal sign (=). Table 5-6 Parameters in the Partitioning Configuration File Parameter Name Explanation Value Default Example table_names Name of the table to partition. Start the table name with the prefix RPT_. Should not be duplicated across sections. RPT_LUR no_of_partition period scheduler_time Number of partitions to create in the database. Number of days that the records are retained in each partition. Time at which to trigger the daily partitioning event scheduler in the database Hour and minutes. Format: HH:MM 23:

88 Configuring Partitioning Chapter 5 Managing Databases and the Comma-Separated Value Repository To create and view partitions, run the partitions.sh script: ~scmscm/scripts/partitions.sh [ --create ] [ --non_partitioned ] [ --partitioned ] [--partitions [table_name]] [events][ --help ] Table 5-7 lists the partitions.sh options. Table 5-7 partitions.sh Options Option --create --non_partitioned --partitioned --partitions [table_name] --events --help Description Create partitions for database tables that are not partitioned. Show details for database tables that are not partitioned. Show details for database tables that are partitioned. Show all the table partitions details. [table_name] specifies the table to be displayed. Show the detail of the configured partitioning events Display the above options Perform the following steps to enable the partitioning feature: 1. Log in as the scmscm user. 2. Change the directory to ~scmscm/scripts/. 3. Customize the ~scmscm/cm/config/dbpacks/mysql/4.0.20/partitions/partitions.conf file based on your requirements. 4. Run the ~scmscm/scripts/loadsqlprocedures.sh --par script to load the required support files for partitioning. 5. Run the ~scmscm/scripts/partitions.sh --create to create the partitioning. Ensure that the periodic deletion process is disabled in the Collection Manager when you enable the partitioning feature. 5-12

89 Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the Oracle Database During the creation of the partitions, the partitions.sh script performs the following actions: Enables the event scheduler on the MySQL server. If any of the tables (table_names) listed in the configuration file (partitions.conf) are not available in the database, partitioning is ignored for that particular table. The remaining tables are processed. When the MySQL server is restarted after the partitioning, the event scheduler is disabled. To ensure the event scheduler remains enabled through a server restart, add event_scheduler = 1 to the [mysqld] section of the my.cnf file. If any missing table can be created in the database later, run the partitions.sh script to create new partitioning. You can apply the same event scheduler for the new partitioning in each configuration section. This does not disturb the existing partitioning. If any new tables are created in the database, define new configuration sections for these new tables and then run the partitions.sh script. This does not disturb the existing partitioning. Although the partitioning event scheduler runs every day, it creates or deletes a partition based on the no_of_partition and period configuration parameters. Even if the server is restarted, the partitioning event scheduler is not disturbed. Managing the Oracle Database The tasks of managing the Oracle database include: Oracle Support with Minimum Privileges Oracle Support with Minimum Privileges The following are the mandatory privileges that are to be granted to the CM-DB user: Create Session, Create Table, Create Procedure, Create Table Space, Unlimited Table Space, Create Index, Execute Any Procedure. The following is the syntax of mandatory privileges: grant CONNECT,CREATE TABLE,CREATE TABLESPACE,CREATE PROCEDURE,UNLIMITED TABLESPACE,CREATE ANY INDEX, EXECUTE ANY PROCEDURE to pqb_admin Connect to the Oracle DB as SYSTEM/SYSDBA, log in to the Oracle server with specific schema (apricot/avocado), and grant the privileges listed here. 5-13

90 Aggregating Database Data Chapter 5 Managing Databases and the Comma-Separated Value Repository Aggregating Database Data You can aggregate database data according to different time periods and store data in the same database tables. Both raw and aggregated data are stored in order to maintain compatibility with the reporting tools that pull this data to generate analysis reports. All the reports that process raw data for display can also process the aggregated data because the data structure is the same. An input date parameter specifies the data records that should be processed. These records are stored per time period. For xur (where x is a particular type of Usage RDR) tables, hourly aggregation is performed, starting from the input date value through the following 14 days. Thereafter, aggregation occurs daily. For HTTP, video, and NUR VSA tables, aggregation occurs daily. During the aggregation process, table rows are aggregated per time period. These values are placed in a temporary table. The source records are then removed to allow insertion of the aggregated records. The aggregation feature is supported in the following tables: RPT_LUR RPT_PUR RPT_VLUR RPT_MALUR RPT_SUR RPT_SPAM RPT_ZUR RPT_TOP_HTTP_DOMAINS RPT_TOP_VIDEO_DOMAINS RPT_TOP_HTTP_HOSTS RPT_TOP_VIDEO_HOSTS RPT_TOP_APN RPT_TOP_DEVICE_TYPE RPT_TOP_NETWORK_TYPE RPT_TOP_USER_LOCATION RPT_TOP_SGSN RPT_TOP_PCF RPT_TOP_HOME_ AGENT RPT_TOP_MEID Caution The actual table records are deleted from the table based on the aggregation range values. If both periodic deletion and the aggregation process are enabled, you can increase the number of days for which the records are retained. 5-14

91 Chapter 5 Managing Databases and the Comma-Separated Value Repository Configuring Aggregation Configuring Aggregation The ~scmscm/cm/config/aggregation.conf file specifies the values used in the aggregation process. The following is a sample configuration file: [agg_event1] procedure_names = P_RPT_MALUR_AGGREGATE, P_RPT_LUR_AGGREGATE no_of_days = 2 scheduler_time = 23:30 Each section in the aggregation configuration file begins with a unique section name in brackets ([]) starting with agg_event. The file contains the parameters shown in Table 5-8. Separate the parameters and their values with an equal sign (=). Table 5-8 aggregation.conf Parameters Parameter Name Explanation Value Default Example procedure_names SQL procedure name to aggregate. no_of_days scheduler_time Number of days of raw data to be retained in the table. The time at which to trigger the daily aggregation event scheduler. Any aggregation procedure name present in the database configuration location. Sybase: ~scmscm/cm/config/dbpacks/ sybase/ase12.5.1/aggregation/ MySQL: ~scmscm/cm/config/dbpacks/ mysql/4.0.20/aggregation/ Oracle: ~scmscm/cm/config/dbpacks/ oracle/9204e/aggregation/ 1 to 31 2 Hour and minutes. Format: HH:MM P_RPT_MALUR_ AGGREGATE 23:30 To create a new aggregation event, run the createaggevents.sh script: ~scmscm/scripts/createaggevents.sh [ --create ] [ --help ] Table 5-9 lists the createaggevents.sh options. Table 5-9 createaggevents.sh Options Option --create --help Description Creates the corresponding aggregation events. Displays these options 5-15

92 Enabling Aggregation in Sybase Chapter 5 Managing Databases and the Comma-Separated Value Repository The Sybase, MySQL, and Oracle databases support aggregation. The following sections describe how to enable the aggregation feature: Enabling Aggregation in Sybase, page 5-16 Enabling Aggregation in MySQL, page 5-16 Enabling Aggregation in Oracle, page 5-17 Use the loadsqlprocedure.sh --agg option to update the database configuration changes. Enabling Aggregation in Sybase Sybase Version 12 and Sybase Version 15 support data aggregation. Sybase should be installed on the same machine to enable the Sybase aggregation. To enable the Sybase aggregation feature, complete the following steps: Step 1 Step 2 Step 3 Step 4 Login as the scmscm user. Run the ~scmscm/scripts/loadsqlprocedures.sh --agg script to load the required support files for aggregation. Create a cron job to run the ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron/run_aggregation.sh script to schedule the aggregation process. Cron job example: crontab -e Update the path and expected time to schedule the job: * * * /<path>/run_aggregation.sh By default, the output of the aggregation process is saved in the Syb_Agg.log file in the ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron directory. For details on the aggregation procedure, see the ~scmscm/cm/config/dbpacks/sybase/ase12.5.1/aggregation/cron/ readme_run_aggregation_procedure.txt file. Enabling Aggregation in MySQL MySQL Version and later support data aggregation. To enable the aggregation feature, complete the following steps: Step 1 Step 2 Login as the scmscm user. Run the ~scmscm/scripts/loadsqlprocedures.sh --agg script to load the required support files for aggregation. 5-16

93 Chapter 5 Managing Databases and the Comma-Separated Value Repository Enabling Aggregation in Oracle Enabling Aggregation in Oracle Oracle Version 9, Version 10, and Version 11 support data aggregation. Complete the following steps to enable the aggregation feature: Step 1 Step 2 Log in as the scmscm user. Run the ~scmscm/scripts/loadsqlprocedures.sh --agg script to load the required support files for aggregation. Make sure that the Oracle user has sufficient privileges to alter and execute the aggregation events. For example: grant select,insert,delete,update on sys.job$ to pqb_admin. 5-17

94 Managing the CSV Repository Chapter 5 Managing Databases and the Comma-Separated Value Repository Managing the CSV Repository Run a utility script to manage the repository of CSV files output by Collection Manager. The CSV adapter writes these files to disk for use by a service provider s operational support system (OSS) or by a third-party billing system. To prevent disk overflow, monitor the size of the CSV repository. If the backup parameter is set to true, failure to delete CSV files may result in disk overflow. No CSV files are ever deleted. A third-party application is responsible for managing the CSV files and deleting them as necessary. To successfully invoke this script, the HTTP adapter of Collection Manager must be running. If the adapter is down, an error message is printed. The following sections detail managing the CSV repository: CSV Repository File Structure, page 5-18 Configuring the CSV File Repository, page 5-18 Configuring the Comma Escape, page 5-20 Configuring Escape of Nonprintable Characters, page 5-20 CSV Repository File Structure CSV files are stored in several subdirectories. Each subdirectory is given the number of a Raw Data Record (RDR) tag. RDR tags define the type of the corresponding RDR. Each RDR is stored in the subdirectory whose name matches its RDR tag number. For more information on RDR tags, see the Cisco Service Control Application for Broadband Reference Guide. The CSV files are numbered sequentially (automatically), with separate numbering in each directory. You can change the location of the parent directory by editing the cm.conf file located in the cm/config directory. Configuring the CSV File Repository Use the ~scmscm/scripts/csvconf.sh script to: List the number of RDRs that are currently stored in the repository. Configure the maximum number of CSV files and the maximum permissible number of reports (lines) in each file. Control whether a backup occurs when an old CSV file is about to be overwritten. Control whether each line in a CSV file contains an indication of the IP address of the Cisco SCE that sent this RDR. (By default, this option is off.) Instead of using the ~scmscm/scripts/csvconf.sh script, you can edit the ~scmscm/cm/config/csvadapter.conf file. For changes in this file to take effect, restart Collection Manager. 5-18

95 Chapter 5 Managing Databases and the Comma-Separated Value Repository Configuring the CSV File Repository The same configuration is applied to all the subdirectories in the CSV repository. Setting these parameters does not change existing the CSV files. Parameter value changes only affect the files that are created after the parameters are set. Table 5-10 lists the csvconf.sh options. Table 5-10 csvconf.sh Options Option --list --clear --maxlines=n --maxfiles=m --backups={true false} --recordsource={true false} Description Display the CSV repository contents (the number of RDRs currently stored in the repository). Delete all the files from the CSV repository. (This option deletes all the CSV files, but not the directories in which they are contained.) Set the maximum number of RDRs per CSV file to N (an integer between 1 and 20,000). Set the maximum number of CSV files in each subdirectory to M (an integer between 10 and 10,000.) Enable or disable the backup of old CSV files. Enable or disable the inclusion of the record source in the CSV files. The following example shows how to set the maximum number of CSV files per subdirectory to 1000: >~scmscm/scripts/csvconf.sh --maxfiles=1000 The following example shows how to set the maximum number records per CSV files to 10,000: >~scmscm/scripts/csvconf.sh --maxlines=10000 The following example shows how to delete all the files from the CSV repository: ~scmscm/scripts/csvconf.sh --clear The following example shows how to disable backing up old CSV files in the repository: ~scmscm/scripts/csvconf.sh -backups=false 5-19

96 Configuring the Comma Escape Chapter 5 Managing Databases and the Comma-Separated Value Repository Configuring the Comma Escape When a comma is contained within a field in a CSV file, an escape sequence indicates that the comma does not mark the end of the field. Three escape sequences are supported: Single quotation marks Single quotation marks surround any field that contains one or more commas. There is no special treatment of single quotation marks that currently exist in the RDRs. URL %2C replaces each comma contained within a field. There is no special treatment of such sequences that currently exist in RDRs. Backslash A backslash (\) precedes each comma contained within a field. There is no special treatment of backslashes that currently exist in RDRs. The first two escape methods are compatible with Microsoft Excel. The backslash method is not compatible with Excel, but is retained for backward compatibility. Single quotation marks are used by default. You can change the escape method by modifying the value of the escapemethod attribute. This attribute is located in the ~/scmscm/cm/config/csvadapter.conf file. You must use one of the following values: quote doublequote backslash url escapemethod is set to quote by default. Configuring Escape of Nonprintable Characters The CSV adapter can escape nonprintable characters. Enabling this option incurs a performance penalty for the adapter. By default, the option is disabled. When the option is enabled, a backslash (\) precedes each character that cannot be printed, such as CR and LF, contained within a field. Enable this option in the ~/scmscm/cm/config/csvadapter.conf file. For the changes made to this configuration file to take effect, restart Collection Manager. 5-20

97 CHAPTER 6 Configuring Databases This chapter describes how to configure the Collection Manager to work with your database. The chapter also describes how to use the database infrastructure to extend the Collection Manager functionality. The following sections describe how to configure the Collection Manager to work with your database: Changing Basic Connection Parameters, page 6-1 Generating SQL Code Using the Velocity Template Language, page 6-1 Database Configuration Files, page 6-3 Working Sample, page 6-6 Testing and Debugging, page 6-9 Using the JDBC Framework in Scripts, page 6-10 Scalability Hints for Oracle, page 6-12 Changing Basic Connection Parameters To use an external database with Collection Manager, change the basic connection parameters, such as the IP address and the port on which the database is deployed. To configure these parameters, run the dbconf.sh script. See the Configuring a Database section on page 4-5. Generating SQL Code Using the Velocity Template Language The JDBC Adapter framework uses macros written in the Velocity Template Language (VTL) to generate all SQL code that is passed to the database server. The following sections describe the configuration file that controls the generation process. For more information regarding VTL (which is a part of the Apache Jakarta Project) go to Table 6-1 describes VTL constructs. 6-1

98 Generating SQL Code Using the Velocity Template Language Chapter 6 Configuring Databases Table 6-1 Summary of VTL Constructs Directive Syntax Example Purpose #foreach #foreach ($item in $collection) item is $item #end Iterates over a collection, array, or map. #if... #else... #elseif #if ($order.total == 0) No charge #end Conditional statement. #parse #parse("header.vm") Loads, parses, and includes the specified template in the generated output. #macro #macro(currency $amount) ${formatter.currency($amount)} #end Defines a new directive and any required parameters, if any. The result is interpreted when used later in the template. #include #include("disclaimer.txt") Includes the specified file, as is, in the generated output. #set #set ($customer = ${order.customer}) Assigns a value to a context object. If the context object does not exist, it is added; otherwise, it is overwritten. #stop #if ($debug) #stop #end Stops template processing. 6-2

99 Chapter 6 Configuring Databases Database Configuration Files Database Configuration Files When you initialize the database access framework, the first file that the database access framework searches for is main.vm, which contains definitions or pointers to all the required database SQL definitions. The location used to search for this file depends on the dbpack used in Collection Manager. A dbpack is a collection of configuration files pertaining to a specific database installation. The adapter (according to its configuration file) selects the dbpack. The following code fragment from the jdbcadapter.conf file configures the Collection Manager to work with an Oracle dbpack: db_template_dir = dbpacks/oracle/9204e/ db_template_file = main.vm The directory location is interpreted relative to the main Collection Manager configuration directory (~scmscm/cm/config). To make the configuration more modular, the main.vm file generally points to other files; however, this mechanism is not strictly necessary. The files can contain arbitrary definitions that can be used later, for example, in scripts. Some definitions are mandatory because the JDBC adapter uses them for its operation. These definitions are listed in Table 6-2. Table 6-2 Mandatory VM Definitions Object Name $table.sql.droptable $table.sql.createtable $table.sql.createindexes $table.sql.insert $table.sql.metadataquery $dbinfo.driverjarfile $dbinfo.driver $dbinfo.cmdseparator $dbinfo.url $dbinfo.connoptions Mandatory Definition For each table, these settings control how SQL is generated for the indicated operation Location and class name for the JDBC driver Pattern used to separate multiple SQL statements URL for connecting to the database Various connection properties Some objects representing Collection Manager configuration in the VTL parsing context can be used in the templates. These objects are described in the following sections: Context Objects, page 6-4 Application Configuration, page

100 Context Objects Chapter 6 Configuring Databases Context Objects The tables Object The dbinfo Object Before any Collection Manager components (for instance, a TA or JDBC adapter, or a script), load and parse the VM templates, Collection Manager initializes the parsing context with the following Java objects: tables dbinfo tools The tables object describes application-related database configuration, such as the structure of RDRs that are stored in the database, the structure of the database tables and where they are stored, and the structure of any other database tables that Collection Manager uses. The object is an array in which each row represents one of the database tables used by Collection Manager. In each table, the row can contain the following information (note that not all items are relevant to all the tables): Logical name Physical name RDR tag associated with this table List of fields/columns in this table, with the following attributes for each: Field ID Field name Field native type Free-form field options List of indexes for this table, with the following attributes for each: Index name Names of columns indexed Free-form index options The contents of the tables object can be inspected or manipulated when loading the templates. The tables object is initialized using the application-specific XML configuration file. See the Application Configuration section on page 6-5. The dbinfo object describes configuration that is specific to the database, such as the parameters and the SID or schema that is used when opening a database connection. The object holds database-specific configuration options. The object contains the following information: The name of the JDBC class that is used as a driver for this database The name of the JAR file that contains the driver The location of the database expressed as a JDBC URL Free-form JDBC connection options, such as authentication data (user and password)the 6-4

101 Chapter 6 Configuring Databases Application Configuration The tools Object The tools object is a container for several utility methods that you can find useful when developing templates or manipulating the context data structures. Invoke the object methods by using the $tools.method(arg1,..., argn), where method is the name of the method. The methods of the tools object are listed in Table 6-3. Table 6-3 Methods of the tools Object Method Name and Arguments gettablebyname (alltables, name) gettablebydbtabname (alltables, name) assignparams (sql, list_of_args) collapsewhitespace() Function Locates the database table object whose logical name corresponds to name. Locates the database table object whose physical name corresponds to name. Replaces question mark characters in the SQL string with consecutive elements from the list_of_args parameter, represented as a String. This method is useful if you are working with templates that create SQL insert statements using the JDBC PreparedStatement string as a base. Converts all instances of more than one consecutive white-space character to one space, and trims the beginning and ending white space. This method can be useful for databases that require SQL with a minimum number of newline and other white-space characters. (Sybase and Oracle do not require this conversion.) For an example that demonstrates how to use these tools, see the Using the JDBC Framework in Scripts section on page Application Configuration All application-related configuration is contained in one file (dbtables.xml), which includes the following items: Name and version of the application Name and properties of each database table, and specifically, the structure of application RDRs that are stored in database tables In each database table: Names and native types of the table and RDR fields Names and properties of the table indexes This information is used primarily to populate the tables object in the template parsing context. See the The tables Object section on page

102 Working Sample Chapter 6 Configuring Databases Working Sample The main.vm file can contain references to other velocity template (VM) files to support modularization (see the Database Configuration Files section on page 6-3). The names of these other files are arbitrary, except for the VM_global_library.vm file, which name is predetermined. Place macros that must be defined in this file to ensure that they are loaded at the right time. For details about this special file, see the Velocity User Guide. The following sample illustrates the contents of main.vm for an Oracle setup: #parse ('dbinfo.vm') #foreach ($table in $tables) #set ($table.sql.droptable = "#parse ('drop_table.vm')") #set ($table.sql.createtable = "#parse ('create_table.vm')") #set ($table.sql.createindexes = "#parse ('create_indexes.vm')") #set ($table.sql.insert = "#parse ('insert.vm')") #set ($table.sql.metadataquery = "#parse ('metadata.vm')") #end In this sample, the mandatory database and SQL definitions (see Table 6-2) are moved to separate files that are loaded and parsed using the #parse directive. The following sections list the possible contents for the various files in the Oracle dbpack. Some of the definitions use macros that are defined in the VM_global_library.vm file. This file contains all macro definitions that any template uses. Macro Definitions, page 6-6 dbinfo Configuration, page 6-7 SQL Definitions, page 6-7 Macro Definitions The following example shows definitions for the mapping between native types and SQL types, and utility macros, such as the optcomma macro, which inserts a comma between successive elements of lists: #macro (optcomma)#if ($velocitycount >1),#end#end #macro (sqltype $field) #set ($maxstringlen = 2000) #if ($field.type == "INT8") integer #elseif ($field.type == "INT16") integer #elseif ($field.type == "INT32") integer #elseif ($field.type == "UINT8") integer #elseif ($field.type == "UINT16") integer #elseif ($field.type == "UINT32") integer #elseif ($field.type == "REAL") real #elseif ($field.type == "BOOLEAN") char(1) #elseif ($field.type == "STRING") varchar2(#if($field.size <= $maxstringlen)$field.size #else $maxstringlen #end) #elseif ($field.type == "TEXT") long #elseif ($field.type == "TIMESTAMP") date #end #end 6-6

103 Chapter 6 Configuring Databases dbinfo Configuration dbinfo Configuration In the following code sample, note that the only required fields are the URL and connection options (for authentication). Blank lines in the code separate the code into distinct fields for readability and to ease later configuration changes. #set ($dbinfo.driver = "oracle.jdbc.oracledriver") #set ($dbinfo.driverjarfile = "ojdbc14.jar") #set ($dbinfo.options.host = "localhost") #set ($dbinfo.options.port = "1521") #set ($dbinfo.options.user = "pqb_admin") #set ($dbinfo.options.password = "pqb_admin") #set ($dbinfo.options.sid = "apricot") #set ($dbinfo.url = "jdbc:oracle:thin:@$dbinfo.options.host:$dbinfo.options.port:$dbinfo.options.sid") #set ($dbinfo.connoptions.user = $dbinfo.options.user) #set ($dbinfo.connoptions.password = $dbinfo.options.password) ## the vendor-specific piece of SQL that will return the current ## date and time: #set ($dbinfo.options.getdate = "sysdate") SQL Definitions Code for Dropping a Table The following sections provide SQL definitions: Code for Dropping a Table, page 6-7 Code for Creating a Table, page 6-7 Code for Creating Indexes, page 6-8 Code for Insert, page 6-8 Code for Metadata Query, page 6-8 The following code sample shows how to drop a table using normal SQL syntax: drop table $table.dbtabname Code for Creating a Table The following code sample shows how to create a table using normal SQL syntax. Use this definition for customized database configurations that require special directives to create a table. For example, you can modify the configuration to create the table in a unique tablespace or use table partitioning. create table $table.dbtabname ( #foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end #end) 6-7

104 SQL Definitions Chapter 6 Configuring Databases Code for Creating Indexes The following code shows how to create an index using normal SQL syntax. Use this definition to implement a customized database configuration that requires special directives to create an index. #foreach ($index in $table.indexes) create index $index.name on $table.dbtabname ($index.columns) #end Code for Insert The following code shows how to create a JDBC PreparedStatement that corresponds to a table s structure: insert into ${table.dbtabname} ( #foreach ($field in $table.fields) #optcomma()${field.name} #end) values ( #foreach ($field in $table.fields) #optcomma()? #end) Code for Metadata Query The following code shows how to define a simple query that gets the table metadata (column names and types). You can use any query that returns an empty result set. select * from ${table.dbtabname} where 1=0 6-8

105 Chapter 6 Configuring Databases Testing and Debugging Testing and Debugging As you develop a set of templates for your database, you can see the results of parsing. To invoke such parsing, the JDBC adapter supports direct invocation using ~scmscm/cm/bin/cm, the Collection Manager s main script. The syntax for invocation is: ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.jdbcadapter argument Here, the argument is one of the parameters described in the following sections. You can use this mechanism whether Collection Manager is running or not. Additionally, you can use the query and update execution methods described in the following sections to test the template results with a live database: Parsing a String, page 6-9 Obtaining Complete Debug Information, page 6-9 Parsing a String You can parse any string as a Velocity Template Language (VTL) template, with the complete context in place. The result of parsing is displayed in the standard output. To parse a string, call the adapter with the -parse parameter. The following sample code shows a parsing mechanism (responses are shown in bold): $ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.jdbcadapter -parse 'xxx' xxx $ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.jdbcadapter -parse '$dbinfo.url' jdbc:oracle:thin:@localhost:1521:apricot $ ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.jdbcadapter -parse '$tools.gettablebyname($tables, "LUR").sql.createTable' create table RPT_LUR ( TIME_STAMP date,record_source integer,link_id integer,generator_id integer,glbl_usg_cnt_id integer,configured_duration integer,duration integer,end_time integer,upstream_volume integer,downstream_volume integer,sessions integer,seconds integer,concurrent_sessions integer,active_subscribers integer,total_active_subscribers integer,ip_type integer Obtaining Complete Debug Information To see a dump of all the contents of the tables and dbinfo structures as created by the templates, use the -debug parameter. The -debug parameter displays, in standard output format, a detailed view of all the fields, properties, and options of these structures. 6-9

106 Using the JDBC Framework in Scripts Chapter 6 Configuring Databases Using the JDBC Framework in Scripts You can send arbitrary SQL commands to the database for execution, and view the resulting data. Viewing the results helps you to perform periodic database maintenance, monitor the contents of database tables, and manage extra database tables. To perform an update operation, call the adapter using the -executeupdate parameter. To perform a query and view the results, call the adapter using the -executequery parameter. Viewing and Setting a Cisco SCE Time Zone Offset The following sample of an update operation demonstrates how to change programmatically, the value in a database table that contains the Cisco SCE time zone offset setting. The name of this table is usually JCONF_SE_TZ_OFFSET. However, because a table can be assigned another name, it is referred to here by its logical name TZ. See the listings in the The dbtables.xml File section on page A-2. To avoid the need to first check if the table exists, and then update it, the table is dropped (ignoring the error status if it does not exist), and then re-created. The proper values should then be inserted. Since the table contains a timestamp column, get the current date from the database. This operation is specific to each database vendor; therefore, the following example uses the preconfigured getdate operation that is defined in the templates. the use of the assignparams and gettablebyname tools to generate the SQL: #! /bin/bash this=$0 tablename=tz usage () { cat <<EOF Usage: $this --status - show currently configured TZ offset $this --offset=n - set the offset to N minutes (-1440 <= N <= 1440) $this --help - print this message EOF } query () { ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.jdbcadapter -executequery "$*" } update () { ~/cm/bin/cm invoke com.cisco.scmscm.adapters.jdbc.jdbcadapter -executeupdate "$*" } get_tz () { query 'select * from $tools.gettablebyname($tables, "TZ").dbtabname' } set_tz () { update '$tools.gettablebyname($tables, "TZ").sql.dropTable' update '$tools.gettablebyname($tables, "TZ").sql.createTable' update '$tools.assignparams($tools.gettablebyname($tables, "TZ").sql.insert, [$dbinfo.options.getdate, '$1'])' } case $1 in --status) 6-10

107 Chapter 6 Configuring Databases Viewing and Setting a Cisco SCE Time Zone Offset esac get_tz ;; --help) usage exit 0 ;; --offset=*) n=$(echo $1 egrep 'offset=[-]?[0-9]+$' sed 's/.*=//') if [ "$n" ]; then if [ "$n" -ge a "$n" -le 1440 ]; then set_tz $n &>/dev/null ok=1 fi fi if [! "$ok" ]; then usage exit 2 fi get_tz ;; *) usage exit 3 ;; When an executed query returns a result set, the result set is displayed as standard output in tabular format with appropriate column headers. 6-11

108 Scalability Hints for Oracle Chapter 6 Configuring Databases Scalability Hints for Oracle The following sections demonstrate ways in which to make database handling more scalable for Collection Manager. These methods are specific to Oracle, and are provided to illustrate the possibilities. Using Custom Tablespaces, page 6-12 Using Table Partitioning, page 6-12 Using Custom Tablespaces If you create several tablespaces and want to distribute Collection Manager tables among them, specify the tablespace to be used for each table, in the dbtables.xml file. The following sample code represents the definition for a table (note that the code is in bold): This sample adds the required tablespaces (tspace1 and tspace2) for the index and the table. There is no preconfigured meaning to the tablespace option in Collection Manager. Any new option name can be used. An option name s meaning is derived from its subsequent use in the templates. To create a table in the correct tablespace, modify the create_table.vm code as follows: create table $table.dbtabname ( #foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end #end) #if ("$!table.options.tablespace"!= "") TABLESPACE $table.options.tablespace #end To create an index in its own tablespace, modify the create_indexes.vm code as follows: #foreach ($index in $table.indexes) create index $index.name on $table.dbtabname ($index.columns) #if ("$!index.options.tablespace"!= "") TABLESPACE $index.options.tablespace #end #end Using Table Partitioning To implement rolling partitioning for a particular table on a weekly basis, create a partitioned option for the table in the dbtables.xml file in a manner similar to the example provided in the previous section (see the Using Custom Tablespaces section on page 6-12). After this, augment the create_table.vm code as follows (note that the code is in bold ): create table $table.dbtabname ( #foreach ($field in $table.fields) #optcomma()$field.name #sqltype($field) #if ("$!field.options.notnull" == "true") not null #end #end) #if ("$!table.options.partitioned"!= "") partition by range (timestamp) 6-12

109 Chapter 6 Configuring Databases Using Table Partitioning (partition week_1 values less than (to_date ('01-JAN :00:00','DD-MON-YYYY HH24:MI:SS')), partition week_2 values less than (to_date ('08-JAN :00:00','DD-MON-YYYY HH24:MI:SS')) partition week_3 values less than (to_date ('15-JAN :00:00','DD-MON-YYYY HH24:MI:SS')) partition week_4 values less than (to_date ('22-JAN :00:00','DD-MON-YYYY HH24:MI:SS')) ); #end Because Oracle does not accept nonconstant expression for time boundaries, the values must be hardwired for the time the tables are created. Create a cron job to roll the partitions (delete an old partition and create a new one) on a weekly basis. This cron job runs a script that calls the command-line interface of the JDBC Adapter (as explained in the Using the JDBC Framework in Scripts section on page 6-10) to issue the appropriate alter table drop partition and alter table add partition SQL commands. 6-13

110 Using Table Partitioning Chapter 6 Configuring Databases 6-14

111 APPENDIXA Code Samples This appendix contains samples of files used to configure the Cisco Collection Manager. The following sections includes file samples for the database table configuration file, the DTD file, the RAG adapter configuration file, and the custom adapter configuration file: Application Configuration, page A-2 RAG Adapter Configuration, page A-5 Custom Adapter Configuration, page A-10 A-1

112 Application Configuration Appendix A Code Samples Application Configuration The following sections shows part of the XML file dbtables.xml used to configure the database tables, and the DTD file used to verify the structure of the XML file. The dbtables.xml File, page A-2 The tables.dtd File, page A-3 The dbtables.xml File The following code is a section of the Cisco SCA BB dbtables.xml file: <?xml version="1.0" encoding="iso8859_1"?> <!DOCTYPE dbtabconf PUBLIC "-//P-Cube//Engage DB RDR Configuration 2.1.0//EN" "dbtables.dtd"> <dbtabconf> <fileversion>... </fileversion> <application name="engage" version="2.1"/> <dbtables> <rdr name="sur" dbtabname="rpt_sur" tag=" " createtable="true"> <fields> <field id="1" name="time_stamp" type="timestamp"> <options> <option property="source" value="timestamp"/> </options> </field> <field id="2" name="record_source" type="int32"> <options> <option property="source" value="recordsource"/> </options> </field> <field id="3" name="subscriber_id" type="string" size="64"/> <field id="4" name="package_id" type="int16"/> <field id="5" name="subs_usg_cnt_id" type="uint16"/> <field id="6" name="monitored_object_id" type="int16"/> <field id="7" name="breach_state" type="uint8"/> <field id="8" name="reason" type="uint8"/> <field id="9" name="configured_duration" type="uint32"/> <field id="10" name="duration" type="uint32"/> <field id="11" name="end_time" type="int32"/> <field id="12" name="upstream_volume" type="int32"/> <field id="13" name="downstream_volume" type="int32"/> <field id="14" name="sessions" type="uint16"/> <field id="15" name="seconds" type="uint16"/> <field id="16" name="os_finger_printing" type="string" size="64"/> <!-- FIELD ADDED FOR IPV6 SUPPORT --> <field id="17" name="ip_type" type="uint8"/> </fields> <indexes> <index name="rpt_sur_i1" columns="end_time"> <options> <option property="clustered" value="true"/> <option property="allowduprow" value="true"/> </options> </index> </indexes> </rdr> <rdr name="lur" dbtabname="rpt_lur" tag=" " createtable="true"> A-2

113 Appendix A Code Samples The tables.dtd File <options> <option property="tablespace" value="tspace1"/> </options> <fields> <field id="1" name="time_stamp" type="timestamp"> <!-- (other field declarations) --> <field id="10" name="downstream_volume" type="uint32"/> <field id="11" name="sessions" type="uint32"/> </fields> <indexes> <index name="rpt_lur_i1" columns="end_time"> <options> <option property="clustered" value="true"/> <option property="allowduprow" value="true"/> <option property="tablespace" value="tspace2"/> </options> </index> </indexes> </rdr> <aggtable name="top_hourly" dbtabname="rpt_tops_period0" aggperiod="0"> <fields> <field id="1" name="record_source" type="int32"/> <field id="2" name="metric_id" type="int8"/> <field id="3" name="subs_usg_cnt_id" type="int8"/> <field id="4" name="time_stamp" type="timestamp"/> <field id="5" name="agg_period" type="int8"/> <field id="6" name="subscriber_id" type="string" size="64"/> <field id="7" name="consumption" type="uint32"/> <field id="8" name="package_id" type="int16"/> <!-- FIELD ADDED FOR IPV6 SUPPORT --> <field id="9" name="ip_type" type="int8"/> </fields> <indexes> <index name="rpt_tops_period0_i1" columns="time_stamp"> <options> <option property="clustered" value="true"/> <option property="allowduprow" value="true"/> </options> </index> </indexes> </aggtable> For each table (either an RDR table, an aggregation table, or an extra table), the fields, indexes, and so on, are listed. A table, an index, or fields can have arbitrary free text options that can be accessed in the templates. The XML file is verified at runtime against a simple DTD, which is reproduced in the following section. The tables.dtd File The following code represents the DTD file used to verify the dbtables.xml definition file: <?xml version="1.0" encoding="iso8859_1"?> <!ELEMENT dbtabconf (fileversion, application, db?, dbtables)> <!ELEMENT fileversion (#PCDATA)> <!ELEMENT application EMPTY> <!ATTLIST application name CDATA #REQUIRED A-3

114 The tables.dtd File Appendix A Code Samples version CDATA #REQUIRED > <!ELEMENT db (options)> <!ELEMENT dbtables (rdr*, aggtable*, table*)> <!ELEMENT table (options?, fields, indexes?)> <!ATTLIST table name CDATA #REQUIRED dbtabname CDATA #REQUIRED createtable (true false) "true" inserttodb (true false) "false" > <!ELEMENT aggtable (options?, fields, indexes?)> <!ATTLIST aggtable name CDATA #REQUIRED dbtabname CDATA #REQUIRED aggperiod CDATA #REQUIRED createtable (true false) "true" > <!ELEMENT rdr (options?, fields, indexes?)> <!ATTLIST rdr name CDATA #REQUIRED dbtabname CDATA #REQUIRED tag CDATA #REQUIRED createtable (true false) "true" inserttodb (true false) "true" > <!ELEMENT fields (field+)> <!ELEMENT field (options?)> <!-- the id attribute below is presumably a numeric index, but it is for future use, we currently don't look at it, as the order is imposed in the XML --> <!ATTLIST field id CDATA #REQUIRED name CDATA #REQUIRED type CDATA #REQUIRED size CDATA #IMPLIED > <!ELEMENT indexes (index+)> <!ELEMENT index (options?)> <!ATTLIST index name CDATA #REQUIRED columns CDATA #REQUIRED create (true false) "true" > <!ELEMENT options (option+)> <!ELEMENT option EMPTY> <!ATTLIST option property CDATA #REQUIRED value CDATA #REQUIRED > You can set the location and name of the DTD and XML files separately for each adapter, in the adapter configuration file. A-4

115 Appendix A Code Samples RAG Adapter Configuration RAG Adapter Configuration The following sections shows the configuration file (ragadapter.conf). It also shows the associated XML files used to configure the RAG adapter. The configuration files of the other adapters are similar to the RAG adapter configuration file. that only the RAG adapter has associated XML files. The ragadapter.conf File, page A-5 ragadapter.xml File, page A-6 The http_turs.xml File, page A-6 The video_turs.xml File, page A-7 The vlink_bw_per_pkg.xml File, page A-8 The vsa_surs.xml File, page A-8 The cdma_surs.xml file, page A-9 The ragadapter.conf File To perform general maintenance of the RAG adapter, use the ~scmscm/cm/config/ragadapter.conf file. The following example represents the RAG adapter configuration file: # # RAGAdapter main configuration file # [config] xml_conf_dir = ~/cm/config/ragadapter domain_ext_filename = ~/cm/config/tlds-alpha-by-domain.txt network_type_filename = ~/cm/config/rat-networktype.txt imei_lookup = false vsa_type=gsm meid_length=8 attr_index=17 attr_shift_pos=0 [housekeeper] interval_sec = 10 [db] operations_timeout = 60 batch_size = 10 transaction_size = 15 commit_interval = 6 blocking_connects = true db_template_file = main.vm db_template_dir = dbpacks/sybase/ase #db_template_dir = dbpacks/sybase/ase #db_template_dir = dbpacks/oracle/9204e #db_template_dir = dbpacks/mysql/ [app] app_conf_file = dbtables.xml app_dtd_file = dbtables.dtd app_conf_dir = apps/scasbb/latest [bench] rate_period_msec = 5000 A-5

116 ragadapter.xml File Appendix A Code Samples ragadapter.xml File The following ragadapter.xml file sample defines the sinks to persist the aggregated data. The sink section specifies the format of the aggregated data store and location to which the data is written (a CSV file or the database). The CSV file definition is located in the <sinks> section under the <csvsink id="csv1"> tag. This definition specifies the output file name and directory. It also specifies the rate at which new files are opened, either every 5 minutes, or when the current file reaches 1000 lines. Each field in the file is enclosed by quotes ("). The database definition is located in the <sinks> section under the <dbsink id="dbsink1"> tag. If the sink is configured as a database, the aggregated data is written to the database configured in Collection Manager. <?xml version="1.0" standalone="no"?> <!--!DOCTYPE ragadapterconf SYSTEM "cm/config/ragadapter/ragadapterconf.dtd"--> <ragadapterconf> <description>rag Adapter basic configuration</description> <config> <aggregations/> <sinks> <csvsink id="csv1" classname="com.cisco.scmscm.adapters.rag.sinks.csvsink" filenameformat="yyyy-mm-dd_hh-mm-ss-sss'.csv'" dirname="~/cm/adapters/ragadapter/csvfiles" maxagesec="300" maxlines="1000" usequotes="true" active="false"/> <dbsink id="dbsink1" classname="com.cisco.scmscm.adapters.rag.sinks.jdbcsink" active="true"/> </sinks> </config> </ragadapterconf> The http_turs.xml File The following sample represents a portion of the http_turs.xml file, which defines aggregations for the incoming HTTP Transaction Usage RDRs (intag=" "). This portion of the http_turs.xml file aggregates RDRs with the RDR tag (outtag="71072") and persists them into the database (sinkid="dbsink1"). The buckets aggregate the data using three HTTP_TUR fields: 7, 18, and 19. These fields correspond to ACCESS_STRING, SERVICE_COUNTER_ID, and PACKAGE_COUNTER_ID. The bucket identifiers are similar to the fields appearing in the SQL GROUP BY clause. The database definition is located in the <sinks> section under the <dbsink id="dbsink1"> tag. The outtag table definition is located in the dbtables.xml tables configuration file. <aggregation id="http TURs by domain, per service, per package" intag=" " outtag="71072" sinkid="dbsink1" period="60" top="500" topid="1" topby="4,5,6,8"> <!-- topby - out index --> <bucketident> <field inindex="7" type="string" index="1" pretransform="com.cisco.scmscm.adapters.rag.transforms.urltodomain"/> <!-- ACCESS_string to domain--> <field inindex="18" type="int" index="2"/> <!-- Service counter id --> <field inindex="19" type="int" index="3"/> <!-- Package counter id --> <totals> <total set="1"/> <total set="2"/> A-6

117 Appendix A Code Samples The video_turs.xml File <total set="3"/> <total set="1,2"/> <total set="1,3"/> <total set="2,3"/> <total set="1,2,3"/> </totals> </bucketident> <closures/> <accumulators> <field inindex="0" type="int" kind="agg_period" index="0"/> <field inindex="0" type="long" kind="count" index="4"/> <field inindex="15" type="long" index="5"/> <field inindex="16" type="long" index="6"/> <field inindex="13" type="long" index="7" unitsfactor="1000"/> <field inindex="0" type="long" kind="uniq_count" index="8"/> <field inindex="6" type="long" kind="rank" index="9"/> <field inindex="4" type="long" kind="rank" index="10"/> <field inindex="8" type="long" kind="rank" index="11"/> </accumulators> <monitors/> </aggregation> The video_turs.xml File The following sample represents a section of the video_turs.xml file, which defines aggregations for the incoming VIDEO Transaction Usage RDRs (intag=" "): <aggregation id="video TURs by domain, per service, per package" intag=" " outtag="71074" sinkid="dbsink1" period="60" top="500" topid="1" topby="4,5,6,8,12"> <!-- topby - out index --> <bucketident> <field inindex="7" type="string" index="1" pretransform="com.cisco.scmscm.adapters.rag.transforms.urltodomain"/> <!-- ACCESS_string to domain--> <field inindex="18" type="int" index="2"/> <!-- Service counter id --> <field inindex="19" type="int" index="3"/> <!-- Package counter id --> <totals> <total set="1"/> <total set="2"/> <total set="3"/> <total set="1,2"/> <total set="1,3"/> <total set="2,3"/> <total set="1,2,3"/> </totals> </bucketident> <closures/> <accumulators> <field inindex="0" type="int" kind="agg_period" index="0"/> <!-- agg period --> <field inindex="0" type="long" kind="count" index="4"/> <!-- SESSIONS counter --> <field inindex="15" type="long" index="5"/> <!-- SESSION_UPSTREAM_VOLUME --> <field inindex="16" type="long" index="6"/> <!-- SESSION_DOWNSTREAM_VOLUME --> <field inindex="13" type="long" index="7" unitsfactor="1000"/> <!-- MILLISEC_DURATION translated to seconds--> <field inindex="0" type="long" kind="uniq_count" index="8"/> <!-- UNIQ SUBSCRIBER --> <field inindex="6" type="long" kind="rank" index="9"/> <!-- SESSION_DOWNSTREAM_VOLUME RANK--> <field inindex="4" type="long" kind="rank" index="10"/> <!-- SESSIONS RANK--> <field inindex="8" type="long" kind="rank" index="11"/> <!-- UNIQ_SUBS RANK--> <field inindex="0" type="long" kind="bit_rate" index="12"/> <!-- BIT_RATE--> <field inindex="12" type="long" kind="rank" index="13"/> <!-- BIT_RATE RANK--> A-7

118 The vlink_bw_per_pkg.xml File Appendix A Code Samples </accumulators> <monitors/> </aggregation> The vlink_bw_per_pkg.xml File The following sample represents a section of the vlink_bw_per_pkg.xml file, which defines aggregations for the uplink and downlink fields in the Subscriber Usage RDRs (intag=" "): <aggregation id="nur's by VLINK UPSTREAM and subs usage counter" intag=" " outtag="71080" sinkid="dbsink1"> <bucketident> <field index="1" type="string"/> <field index="12" type="int"/> <!-- Check the type --> </bucketident> <closures/> <accumulators> <!-- duration=6, up=8, down=9, sessions=10, seconds=11 (not to use) --> <field index="6" type="long"/> <field index="8" type="long"/> <field index="9" type="long"/> <field index="10" type="long"/> <field index="11" type="long"/> </accumulators> <monitors> <timeoutmonitor action="checkpoint" maxsec="900" active="true"/> </monitors> </aggregation> The vsa_surs.xml File The following sample represents a section of the vsa_surs.xml file, which defines aggregations for the VSA fields present in the Subscriber Usage RDRs (intag=" "): <aggregation id="vsa SURs by device type, per service" intag=" " outtag="71076" sinkid="dbsink1" period="60" top="500" topid="1" topby="3,4,5"> <!-- topby - out index --> <bucketident> <field inindex="19" type="string" index="1" pretransform="com.cisco.scmscm.adapters.rag.transforms.imeitodevicetype" /> <field inindex="2" type="int" index="2" /> <!-- Service usage counter id --> <totals> <total set="1" /> <total set="2" /> <total set="1,2" /> </totals> </bucketident> <closures /> <accumulators> <field inindex="0" type="int" kind="agg_period" index="0" /> <field inindex="8" type="long" index="3" /> <field inindex="9" type="long" index="4" /> <field inindex="0" type="long" kind="uniq_count" index="5" /> <field inindex="4" type="long" kind="rank" index="6" /> <field inindex="5" type="long" kind="rank" index="7" /> </accumulators> <monitors /> </aggregation> A-8

119 Appendix A Code Samples The cdma_surs.xml file The cdma_surs.xml file The following sample represents a section of the Cdma_SURs.xml file, which defines aggregations for the VSA fields present in the Subscriber Usage RDRs (intag=" "): <aggregations> <aggregation id="cdma SURs by device type, per service" intag=" " outtag="71082" sinkid="dbsink1" period="60" top="-1" topid="1" topby="3,4,5"> <!-- topby - out index --> <bucketident> <field inindex="26" type="string" index="1" pretransform="com.cisco.scmscm.adapters.rag.transforms.meidtodevicetype" /> <!-- Need to identify device type--> <field inindex="2" type="int" index="2" /> <!-- Service usage counter id --> <totals> <total set="1" /> <total set="2" /> <total set="1,2" /> </totals> </bucketident> <closures /> <accumulators> <field inindex="0" type="int" kind="agg_period" index="0" /> <!-- agg period --> <field inindex="8" type="long" index="3" /> <!-- SESSION_UPSTREAM_VOLUME --> <field inindex="9" type="long" index="4" /> <!-- SESSION_DOWNSTREAM_VOLUME --> <field inindex="0" type="long" kind="uniq_count" index="5" /> <!-- UNIQ SUBSCRIBER --> <field inindex="4" type="long" kind="rank" index="6" /> <!-- RANK VOLUME--> <field inindex="5" type="long" kind="rank" index="7" /> <!-- UNIQ_SUBS RANK--> </accumulators> <monitors /> </aggregation> A-9

120 Custom Adapter Configuration Appendix A Code Samples Custom Adapter Configuration The Custom Adapter allows users to configure specific fields in a CSV file for a particular RDR. The ~scmscm/cm/config/customadapter/custom.xml file contains this configuration: <customadapter> <description>customized CSV Adapter</description> <config> <rdrs> <rdr tag=" " sinkid="csv1"> <fields> <field outindex="0" name="rdr_tag" type="string" constant=" " /> <field outindex="1" name="link_id" type="string" rdrindex="0" /> <field outindex="2" name="gen_id" type="string" rdrindex="1" /> <field outindex="3" name="svc_id" type="string" rdrindex="2" /> <field outindex="4" name="upstream_vol" type="long" rdrindex="6" /> <field outindex="5" name="downstream_vol" type="long" rdrindex="7" /> <field outindex="6" name="total_vol" type="long" expr="(f6+f7)" /> <!--<field outindex="7" name="sub_const" type="string" expr="f0+fc" constant="hello"/>--> </fields> </rdr> </rdrs> <sinks> <csvsink id="csv1" classname="com.cisco.scmscm.adapters.custom.sinks.csvsink" filenameformat="yyyy-mm-dd_hh-mm-ss-sss'.csv'" dirname="~/cm/adapters/customadapter/csvfiles/ " maxlines="1000" active="true" /> </sinks> </config> </customadapter> These configured RDR TAG IDs should be added in the ~scmscm/cm/config/queue.conf file under the [bill-custom] section. The sink information should also be configured in the ~scmscm/cm/config/customadapter/custom.xml file for the Custom Adapter to store the customized RDR records. The sink IDs should be unique for each RDR. Table A-1 defines the fields in the custom.xml file. Table A-1 custom.xml File Fields Fields outindex name type rdrindex expr (ex:- expr Definition Refers to the column index of the csv file. Column name of the csv file. Refers to the Data Type for the corresponding column Refers to the rdr fields that need to be processed. Performs Arithmetic Expressions between RDR Fields and with constant values. "(F6+F7)"à Perform Addition operation for the rdr field 6 and rdr Field 7 and store the result in the corresponding outindex column A-10

121 Appendix A Code Samples Custom Adapter Configuration Table A-1 custom.xml File Fields Fields expr constant Definition (F0+FC) à Add rdr Field 0 with a constant value, FC refers to the constant value that needs to be added). Stores the constant value in the corresponding outindex column. A-11

122 Custom Adapter Configuration Appendix A Code Samples A-12

123 APPENDIXB Configuration File Descriptions This appendix describes each of the configuration files used by Collection Manager along with the configuration parameters in each file. Each configuration file is divided into sections. A heading identifies each section, for example, [config]. You can comment out any parameter that is not required in your configuration by placing a # at the start of the line. If you modify the configuration file, restart Collection Manager to enable the changes to take effect. Caution The configuration files supplied with Collection Manager software installation contain tested valid settings. You are not expected to change the settings. Use care when modifying the settings. B-1