Resonate Central Dispatch User Guide. Version 3.3

Transcription

1 Resonate Central Dispatch User Guide Version 3.3

2 Copyright Resonate, Inc. All rights reserved. Resonate Incorporated and its licensors retain all ownership rights to the Central Dispatch computer programs (hereinafter collectively called Resonate Software) and their documentation. Use of Resonate Software is governed by the license agreement accompanying your original media. Information in this book is subject to change without notice. Companies, names, and data used in examples herein are fictitious unless otherwise noted. Resonate is a registered trademark of Resonate, Inc. The Resonate logo, Resonate Central Dispatch, Resonate Commander, and Resource-based Scheduling are trademarks of Resonate, Inc. Sun, the Sun logo, Sun Microsystems, Solaris, SPARC, Java, and all Java-based trademarks and logos, are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. UNIX is a registered trademark of the Open Group. RS/6000, Power2, and PowerPC are trademarks, and AIX and IBM are registered trademarks of International Business Machines Corporation. HP-UX and Hewlett Packard are registered trademarks of Hewlett Packard Company. Microsoft, Windows, and Windows NT are registered trademarks of Microsoft Corporation. Linux is a registered trademark of Linus Torvalds. All other brand product names mentioned herein may be the trademarks or registered trademarks of their respective companies. Resource-based Scheduling technology is protected under U.S. Patent 5,774,660. Version: 3.3 Date: December 11, 2000 Part Number: a

3 Table of Contents Preface Before you install Central Dispatch... vii Who should read this book... vii How this book is organized... vii Typographic Conventions... viii Contacting Resonate... ix 1 Introducing Central Dispatch Overview Central Dispatch architecture RXP Dispatch Manager Central Dispatch agents Nodes: schedulers and servers Scheduler nodes Server nodes Affiliated server nodes Example scenario VIPs Advantages to using Central Dispatch Multihoming Scheduling Round-Robin scheduling Resource-based Scheduling Content distribution strategies Server-based distribution Indexed distribution Load Balancing Thresholds Class of service thresholds Protection from attacks Resonate Central Dispatch User Guide iii

4 Denial of service attacks Bandwidth allocation denial Man Pages and Help Files Summary Using Central Dispatch Using Dispatch Manager Starting Dispatch Manager Views Setting up your site Creating VIPs Designating schedulers Choosing scheduling policies Choosing a load-balancing policy Shadow Scheduling Address changes and schedulers Affiliated servers and master agents Affiliated servers and scheduling Half NAT Full NAT Configuring groups of nodes Setting configuration options Using live operations Licensing the site Starting the site Changing a running site Verifying your changes Exporting and importing a configuration Exporting a configuration Importing a configuration Disabling or removing nodes Stopping the site Starting and stopping Central Dispatch Agents UNIX Windows NT and Windows Solaris UDP Buffers AIX Socket Buffers Removing a server from service Auto site restart Exiting Dispatch Manager iv Resonate Central Dispatch User Guide

5 Useful environment variables Set up protection from attacks Denial of service attacks Bandwidth Allocation Denial Monitoring Your Site Overview Connecting to a running site Displaying site statistics Viewing node activity Viewing VIP activity Messages and agent logs Viewing messages Filtering messages Managing messages and agent logs Agent log file format Setting up alarms Using the pager alarm script Creating alarms Testing alarms Modifying alarms Deleting alarms Windows NT and Windows 2000 Performance Monitor Using Scheduling Rules Rule components COS Groups and Thresholds Failover servers Kinds of scheduling Service-based scheduling VIP-based scheduling Port-based scheduling Path-based scheduling Cookie/CGI-based scheduling Persistent service scheduling SSL3 scheduling How Central Dispatch chooses a rule Specifying scheduling rules Changing and deleting scheduling rules Resonate Central Dispatch User Guide v

6 Loading and saving scheduling rules Saving rules for use with Commander Modifying Firewalls Overview Protocols General requirements Packet screening Forward the Central Dispatch port Forward RXP protocols Forward VIPs Network Address Translation Proxy filters Permitting Commander Monitor testing Administering From the Command Line Description Syntax Parameters Exit status and output Examples A Error Messages Overview... A-1 Messages... A-2 Agent Start / Stop and Normal Configuration... A-3 Node Health Reporting... A-4 Master Agent Negotiation... A-4 Rule Critical Errors... A-5 Agent Critical Errors... A-5 Communication Critical Errors... A-7 Out of Kernel Memory Critical Errors... A-9 Dispatch Manager Critical Errors... A-11 Glossary Index vi Resonate Central Dispatch User Guide

7 Preface Central Dispatch is a distributed server management system for Internet and intranet sites. Central Dispatch enhances sites by offering exceptional traffic control, availability, and administrative flexibility. Before you install Central Dispatch See the Resonate Central Dispatch Resonate Commander Site Preparation Guide to see the system requirements and procedure for installing Central Dispatch. Who should read this book To use Central Dispatch you should be a system administrator with a working knowledge of TCP/IP networking and the operating systems your machines use. How this book is organized This book describes how the Central Dispatch software works and how to use it to manage complex internet sites. Chapter 1 introduces the product and describes its components. Chapter 2 describes how to set up and start a Central Dispatch site using a simple configuration. This chapter also Resonate Central Dispatch User Guide vii

8 describes the administrative options available for a running site. Chapter 3 describes the monitoring options available on a running site. Chapter 4 explains how to configure Resource-based Scheduling rules. Chapter 5 tells you how to configure a firewall (if you have one) to work with Central Dispatch. Chapter 6 explains how to administer Central Dispatch from a shell or command prompt. The Glossary defines many important terms that relate to Central Dispatch. Typographic Conventions Computer output and file names appear this font. Commands, URLs, and other text that you type appears in this bold font. Variables appear in italics, like this. Important new terms appear in bold, like this. Elements from the graphical user interface, such as buttons and dialogs, also appear bold, like this. viii Resonate Central Dispatch User Guide

9 Contacting Resonate You can reach Resonate Technical Support as follows: Web Site Telephone Toll free RESONATE ( ) International Resonate Technical Support staff responds to telephone and comments 7 days a week, 24 hours a day. For Resonate Sales call Resonate welcomes your comments. Send comments by to: comments@resonate.com Resonate Central Dispatch User Guide ix

10 x Resonate Central Dispatch User Guide

11 CHAPTER 1 Introducing Central Dispatch Overview This chapter describes the product architecture of Central Dispatch. It also describes RXP (Resonate Exchange Protocol), Resonate Dispatch Manager, and the Central Dispatch agents. Central Dispatch is a distributed server management system for Internet and intranet sites. Central Dispatch enhances sites by offering exceptional traffic control, availability, and administrative flexibility. Central Dispatch enables multiple Internet servers to act as a single, scalable, reliable, and easily-managed Internet server system. Servers in a Central Dispatch site are accessed through one or more VIPs (Virtual IP addresses) and appear to clients as a single Internet site. Servers can be co-located on one or more IP subnets or distributed geographically, providing support for heterogeneous UNIX and Windows environments, multiple departmental networks, and firewall architectures. Content can be replicated on multiple servers for highest availability or segregated by file type (for example, CGI scripts, graphics files, HTML pages, and so on) or by subject (for example, */memberlogin/*) to minimize response times for client requests. Central Dispatch supports both Resource-based Scheduling and round-robin scheduling. For complex sites and high-traffic sites, Resource-based Scheduling is most efficient, flexible, and provides the best control. Resonate Central Dispatch User Guide 1-1

12 Introducing Central Dispatch Central Dispatch provides precise control over incoming traffic. Central Dispatch schedulers provide this control by receiving client connection requests and transferring them to the appropriate server. Central Dispatch combines the schedulers and servers (wherever they are located on the network) into one or more Central Dispatch sites.it is these sites that both clients and other servers on the Internet (such as DNS servers) are aware of and refer to by IP addresses (VIPs). Each of the computers that constitute a site (the schedulers and servers) is called a node. The collection of nodes is the Central Dispatch site. Multiple schedulers improve the scalability of Central Dispatch sites. For small to medium-sized sites, the multiple scheduler capability means that no server has to be dedicated as a scheduler. For large sites, additional schedulers can be added easily without the need to split a site. Also, all site resources can be configured and monitored from a single view. A primary scheduler and an backup scheduler can be assigned to each VIP. Each scheduler can map requests to any and all content servers at the site. Note This version of Central Dispatch supports a maximum of 16 schedulers and 64 nodes (schedulers and content servers) in a site. Central Dispatch uses user-defined scheduling rules to match incoming URL requests with servers and ports best suited to respond. The schedulers base their scheduling decisions both on run-time conditions and administrative policies and rules (such as which servers have which kinds of content). If multiple servers are eligible, the loadbalancing policy determines which server is selected based upon current conditions including CPU load, number of open connections, and network latency. The TCP connection hop the schedulers use to transfer connection requests to the selected server is completely transparent to the client and the server neither is aware that the original client request was received by the scheduler, nor that the request was then transferred by the scheduler to the server. 1-2 Resonate Central Dispatch User Guide

13 Introducing Central Dispatch The port mapping feature allows requests for specific ports to be sent to your chosen port. Port load balancing allows Central Dispatch to route client requests to one of a set of ports on a server in round-robin fashion across the available ports. Central Dispatch supports failover if one of the components in a Central Dispatch site stops functioning, a spare or backup component comes online to prevent any interruption of processing. Central Dispatch also supports recovery if a node stops functioning, it is automatically reintegrated into the site when the node starts functioning again. Central Dispatch sites are scalable as use patterns and traffic levels change or increase, you can upgrade the site (adding, changing, or removing resources) without interrupting service. Central Dispatch works transparently with other traffic management solutions, such as Resonate Commander and Resonate Global Dispatch, which provides optimized traffic management across high volume, geographically-dispersed Points-of-Presence. Note For more information about Commander, see the Resonate Commander User Guide. For more information about Global Dispatch, see the Resonate Global Dispatch User Guide. Central Dispatch architecture Central Dispatch runs on each node in your site and consists of: RXP (Resonate Exchange Protocol) module This kernellevel module receives user requests on one or more VIPs and manages the TCP connection hop from scheduler node to server node. Resonate Central Dispatch User Guide 1-3

14 Introducing Central Dispatch Central Dispatch agents The agents unify your site by communicating site information to one another. To configure, monitor, and manage your Central Dispatch site (and Commander), you use Dispatch Manager. This Java application, which is also a component of Central Dispatch, can run on any supported platform that can connect over the network to the nodes in your Central Dispatch site (and to the Commander hosts). Note To learn more about installing and deploying Central Dispatch components, read the Resonate Central Dispatch Resonate Commander Site Preparation Guide. RXP Figure 1-1 illustrates the communication paths between Central Dispatch components (including Resonate Commander) and Internet clients (such as Web browsers or other clients). Bold lines denote client communication with a Central Dispatch site. Bold dashed line denotes the TCP connection hop from the scheduler to the server node selected by the scheduler to service the client request. Dotted lines denote communication between agents. The thin solid line denotes the Dispatch Manager connection to the site and the Commander host. RXP (Resonate Exchange Protocol) is the heart of Central Dispatch. RXP is a kernel-level service that is inserted on each server below the native TCP/IP stack. RXP performs the TCP connection hop and Resource-based Scheduling functions in Central Dispatch. (To learn more, read Resource-based Scheduling on page 1-20.) 1-4 Resonate Central Dispatch User Guide

15 Introducing Central Dispatch Client Commander Dispatch Manager Internet RXP Agent RXP Agent RXP Agent RXP Agent Scheduler Node Server Node Server Node Server Node Figure 1-1 Communication among Central Dispatch components and Internet clients Dispatch Manager Dispatch Manager is a Java application that lets you configure, monitor, and administer a Central Dispatch site. (It also lets you configure and administer Commander. To learn how, read the Resonate Commander User Guide.) Dispatch Manager runs on any supported platform that can connect using TCP/IP to every node in your Central Dispatch site. You can also use Dispatch Manager remotely. For example, if you need to check on a Central Dispatch site when you are out of the office, you can run Dispatch Manager on a laptop computer, connect to the Resonate Central Dispatch User Guide 1-5

16 Introducing Central Dispatch Figure 1-2 Dispatch Manager welcome screen site with a modem using PPP (Point-to-Point protocol), view the status of the Central Dispatch nodes, and if needed, reconfigure the site. In administration mode you can use Dispatch Manager to: Configure the site, including add and delete nodes, assign VIPs to nodes, and create scheduling rules. Start and stop Central Dispatch, and apply or discard configuration changes. Reschedule failed servers. Display statistics, including hits per second, number of open connections, CPU load, and network latency. Set up alarms that send to or page the site administrator (or anyone else) when specified types of events occur. 1-6 Resonate Central Dispatch User Guide

17 Introducing Central Dispatch View messages and manage the message log. Configure and administer Resonate Commander. Note For more information about Resonate Commander, read the Resonate Commander User Guide. In monitor mode you can use Dispatch Manager to view what is going on at the site. However, you cannot reconfigure, stop, or start the site in monitor mode. To use Dispatch Manager in administration mode, you enter the administration mode account password when prompted for a password; to use Dispatch Manager in monitor mode, you enter the monitor mode password. (For more information about these accounts, read the Resonate Central Dispatch Resonate Commander Site Preparation Guide.) Central Dispatch agents Central Dispatch agents provide the communication and control necessary to bind the individual machines in the site into a unified entity. Agents are daemons (on UNIX) or services (on Windows) that communicate with each other to keep the Central Dispatch site configured properly and available. They also respond to management directives from Dispatch Manager. All agents communicate with one of the agents known as the master agent, which collects statistics and other information from the other agents. The master agent then sends the status and configuration information for the entire site to all the other agents and to Dispatch Manager. Because each agent has all of the site information, any agent can be a master agent. If the current master agent becomes unavailable, one of the other agents automatically becomes the new master agent. Resonate Central Dispatch User Guide 1-7

18 Introducing Central Dispatch The Central Dispatch agent process spawns a child process to do the actual work of the agent, then the parent goes to sleep. The parent wakes up only if it becomes necessary to respawn the child process (should the child process die, for example). Node configuration When you configure a Central Dispatch site and use Dispatch Manager to start the site, Dispatch Manager sends the directive to all the Central Dispatch agents in the site. At each node the agent configures RXP on that node to perform the role of a scheduler, a server, or both. The agent also configures the network interfaces for all VIPs and marks them as UP. Inter-agent communication Each agent periodically sends status information to the master agent. In turn, each agent receives status and configuration information for the entire site from the master agent. You can set the time interval between communications with the heartbeat interval (see Setting configuration options on page 2-27) using Dispatch Manager. The default heartbeat interval works well for most sites. Increase the heartbeat interval for large sites (more than 10 nodes). Agent timeouts If messages sent by the master agent to another agent time out, Dispatch Manager generates the message Agent missed a ping. If an agent times out, its node is considered unresponsive. You can configure this timeout in Dispatch Manager by setting the Heartbeats Until Down parameter (see Setting configuration options on page 2-27). The default value for this parameter works well in most cases. You can increase the value of Heartbeats Until Down if your network is heavily loaded and Dispatch Manager erroneously reports nodes as down. 1-8 Resonate Central Dispatch User Guide

19 Introducing Central Dispatch High availability Central Dispatch monitors the status of all nodes in the Central Dispatch site and implements automatic failover and recovery mechanisms in the event of a node failure. Server failover When a server node becomes unresponsive, the master agent detects this and notifies the scheduler to stop sending requests to that node. If content is replicated across multiple servers and rules are set up accordingly, service continues as the scheduler transfers requests to the remaining eligible servers. Another kind of server failover occurs if a server node responds to a ping, but the application software (such as a Web server) does not respond. In this case, the RXP module transfers the connection to another available server node in the site. When a failed node restarts, it automatically reintegrates into the Central Dispatch site. This means that the scheduler once again can transfer client connection requests to that node using the TCP connection hop. Scheduler failover Central Dispatch agents can ensure that your site remains available even if the primary scheduler becomes unresponsive. Each scheduler license enables you to designate both a primary scheduler and a backup scheduler per VIP. If the VIP becomes unresponsive, the system assumes that the primary scheduler has failed and instructs the backup scheduler to take over and fulfill the request directed to the VIP. You can configure the site so that when a failed primary scheduler comes back up, the master agent either automatically reinstates the primary as the scheduler that handles client connection requests to the VIP or leaves the backup as the acting primary. Note that with multiple primary schedulers in a site, each primary scheduler can have a configurable backup server. A backup scheduler can be an inactive standby server, another primary scheduler, or a Resonate Central Dispatch User Guide 1-9

20 Introducing Central Dispatch content server. The only requirement is that the backup scheduler must be on the same subnet as the primary. Auto site restart If all the nodes in a site fail, such as during a power outage, the auto site restart feature starts up the Central Dispatch site when the machines recover. The feature requires installation of the CDAction component on at least one node in the Central Dispatch site. Resonate recommends setting up primary and backup scheduler nodes with CDAction to act as triggers for auto site restart. Auto site restart is the default state of a Central Dispatch site that has at least one node with CDAction installed. To disable this feature, see Auto site restart on page Communication with Dispatch Manager Because each agent has site-wide configuration and status information, any agent: Can provide full information about the status and configuration of the site to Dispatch Manager Can forward to the master agent a Dispatch Manager directive to start up, reconfigure, or shut down, disseminating the directive throughout the site. Dispatch Manager can therefore connect to any node in the site to manage the site. Affiliated server nodes affiliated server nodes in the site do not have Central Dispatch software installed and may use operating systems that are not supported by Resonate software. All of the agent functions described above are not relevant to an affiliated server node. However, such nodes can receive traffic scheduling services from Central Dispatch. A scheduler does communicate with an affiliated node to determine the number of open connections and network latency Resonate Central Dispatch User Guide

21 Introducing Central Dispatch Here is a summary of information needed to use affiliated servers as content nodes in a Central Dispatch site: No Central Dispatch software needs to be installed on an affiliated server. All of the reporting and communication functions of a Central Dispatch agent are not available for an affiliated server. The scheduling parameters available for an affiliated server are limited to: Open connections Network latency Round robin Weighted round robin An affiliated server cannot be a scheduler. An affiliated server can use half NAT or full NAT as a routing mechanism for TCP/IP traffic. A site that includes affiliated servers cannot have multiple schedulers for the whole site unless the routing mechanism is full NAT. With half NAT, and from a single Central Dispatch site, only one scheduler at a time can connect to an individual affiliated server. Schedulers from different Central Dispatch sites can connect to the same affiliated server. To be able to schedule to an affiliated server, a site with affiliated servers must have at least one node set up as a scheduler node on a Solaris, HP-UX, Windows NT, or Windows 2000 machine. A Linux or AIX machine set up as a scheduler cannot recognize affiliated servers. A site with affiliated servers cannot have a scheduler using a Central Dispatch software from any version earlier than version 3.1. Schedulers must be version 3.1 or later to recognize affiliated servers. A single site can mix affiliated server content nodes (with no agents) and standard Central Dispatch content nodes (with agents). The nodes with agents can be a mix of any platforms supported by Central Dispatch software. Resonate Central Dispatch User Guide 1-11

22 Introducing Central Dispatch The Advanced Options button of the Operations tab on the Site Setup view brings up a dialog box that allows you to set half NAT or full NAT as the routing mechanism for forwarding TCP/IP traffic to an affiliated server. Resonate recommends using full NAT. The dialog box for the Advanced Options button of the Operations tab on the Site Setup view also has a Use Client Route Caching checkbox. Client route caching (for affiliated servers only) uses past information to return packets to a client across the same route. This may improve performance in a CPUbound system. The trade-off is that if a change in network topology changes the route to the client, responses are delayed until the new route is determined. Nodes: schedulers and servers A Central Dispatch site, as shown in Figure 1-3, consists of a set of nodes (computers) that deliver Web pages and other Internet services to clients outside the site anywhere on the Internet. The nodes in a Central Dispatch site need not be on the same LAN; they can be dispersed across LANs, WANs, and firewalls. Clients must be outside the Central Dispatch site. A Central Dispatch site consists of one or more scheduler nodes and one or more content server nodes. A scheduler node receives all incoming requests. After evaluating a request, the scheduler node passes the request to a content node by performing a TCP connection hop (see Figure 1-4). The content node then responds to the request. A server node can be any kind of server, including a Web server that serves up Web pages and the output from programs such as CGI programs. Central Dispatch is compatible with all standard Web server software Resonate Central Dispatch User Guide

23 Introducing Central Dispatch Client Internet Scheduler Server Node Node Central Dispatch Site Figure 1-3 A Central Dispatch site Scheduler nodes Each scheduler node maintains a list of VIPs to which it routes requests. No VIP can be on more than one primary scheduler s list. All incoming requests to a particular VIP are received by one scheduler node. The scheduler determines which server is best suited to respond by evaluating user-specified metrics including CPU load, open connections, network latency, and server weighting parameters. After evaluating a request, the scheduler node passes the request to the selected server node by performing a TCP connection hop (see Figure 1-4). The server node then responds directly to the client request. Resonate Central Dispatch User Guide 1-13

24 Introducing Central Dispatch Note An affiliated server cannot be a scheduler. Schedulers must be on either Solaris, HP-UX, Windows NT, or Windows 2000 platforms to recognize affiliated servers. Using multiple scheduler nodes At a Central Dispatch site, you can configure multiple primary schedulers, each with its own backup scheduler. Each primary scheduler sends requests through its list of VIPs to the most appropriate server. More than one scheduler can route requests to the same server nodes. When there are multiple schedulers in a Central Dispatch site, they all share the same configuration parameters, scheduling rules, and load balancing information (statistics collected on the server node). Central Dispatch s multiple scheduler architecture means that, even in high traffic situations, incoming requests can quickly be routed to the appropriate server. Because each primary scheduler can be assigned a backup scheduler, single points of failure can be eliminated. For more information, read Scheduler failover on page 1-9. Server nodes The server node responds to the client request passed to it by a scheduler. A server node can be any kind of server that communicates using TCP/IP, including a Web server that serves up Web pages, the output from programs (such as CGI programs), FTP servers, and SSL servers. Central Dispatch is compatible with all standard Web server software. You can replicate server node content on multiple servers for the highest availability. You may also want to segregate content by file type or by subject matter. For example, you could have one server node serve up requests for all graphic (.gif) files and another for all HTML 1-14 Resonate Central Dispatch User Guide

25 Introducing Central Dispatch files. You could also have different server nodes store different categories of content. For example, one server node might have information on the arts (/arts), another on business (/business), and so on. For more information, read Scheduling on page Affiliated server nodes Example scenario A Central Dispatch site can include servers using unsupported (by Resonate) operating systems as content server nodes. Such affiliated nodes can receive traffic scheduling services from Central Dispatch even though the nodes have no Central Dispatch software installed. The measurement metrics for an affiliated node are the number of open connections and network latency. The scheduler node and server nodes cooperate to satisfy user requests. The following scenario describes how a Web browser s request for a document is handled by Central Dispatch. The client (Web browser) requests a document by connecting to your Central Dispatch site and sending the URL (Uniform Resource Locator) of the document. The host specified in the URL is resolved through a DNS lookup to the VIP of your Central Dispatch site. The scheduler node for this VIP at the site receives the request, examines the URL, then decides which server node and which port on that server should respond to the request according the scheduling rules you have defined. The scheduler transfers the TCP connection to the selected server node and port. The Web server software on the server node responds directly to the client browser over the connection that the scheduler has transferred to the server node. Figure 1-3 depicts the flow of information between the client and a Central Dispatch site. The line denoting the client request is thinner than the line denoting the response because Web-related network traffic is generally asymmetric clients send small messages containing URLs to servers, whereas servers send larger amounts of Resonate Central Dispatch User Guide 1-15

26 Introducing Central Dispatch Web Browser Request Internet Response TCP Connection Hop Scheduler Node Figure 1-4 TCP connection hop Server Node data to clients in response. Central Dispatch is designed accordingly: low-bandwidth requests go through the scheduler node, whereas highbandwidth responses pass directly from the server nodes to the clients. VIPs You register Central Dispatch site IP addresses with DNS (Domain Name System) servers just as you register any IP address. However, unlike standard IP addresses, the virtual IP addresses (VIPs) denote the entire Web site and not a particular computer. Caution VIPs must not be assigned to any physical computer on the network. They are always assigned to the site as a whole through Dispatch Manager Resonate Central Dispatch User Guide

27 Introducing Central Dispatch Web Client Internet Scheduler Node Figure 1-5 Server Node VIP ( = ) Scheduler and server nodes together create a site At the Central Dispatch site the scheduler nodes (and only the scheduler nodes) listen to network traffic directed to the VIPs. After the scheduler passes the request to a server node, the server node responds to the client using the VIP specified in the client s request as the source address. Figure 1-5 illustrates this process. Because the request and the response have the same address, the user is unaware that multiple systems cooperated to serve the request. The client has the illusion that a single server has responded to the request. Resonate Central Dispatch User Guide 1-17

28 Introducing Central Dispatch Web Client to from Internet Scheduler Node Server Node Figure Central Dispatch site is both destination and source Advantages to using Central Dispatch By using a site to represent the work of a number of nodes, Central Dispatch: Prevents problems associated with client-side caching of individual server IP addresses. Client-side caching of IP addresses can lead to errors if a particular server is unavailable at the time of the next connection attempt Resonate Central Dispatch User Guide

29 Introducing Central Dispatch Ensures that all users can reach the site. Because a VIP identifies the collection of all the server and scheduler nodes in a Central Dispatch site, the site is accessible as long as a scheduler node is running and at least one server node that can be selected by the scheduler according to the current scheduling rules is running. Multihoming When a Central Dispatch site serves multiple logical sites (such as a Web site, an FTP site, and so on), different VIPs differentiate the sites. The same Central Dispatch scheduler node can handle requests for many different VIPs and route requests to the correct servers based on the VIP supplied in the client request. This multihoming feature lets you define multiple logical sites using a single Central Dispatch site. Scheduling At the scheduler node, the actual scheduling mechanism is determined by the scheduling policy. Central Dispatch currently supports two scheduling policies: round-robin and Resource-based. Round-Robin scheduling This scheduling policy has the scheduler node use a simple roundrobin process of rotating assignments among all servers in the site. This type of scheduling: Enables you to get a site up and running quickly. Is useful for testing. Round-robin scheduling requires that each server contains a complete replication of server content. Complete content replication ensures high availability but has certain costs and limitations: Resonate Central Dispatch User Guide 1-19

30 Introducing Central Dispatch As a site gets larger, complete replication becomes increasingly expensive. If licensed software is in use, a license is needed at every server. Resource-based Scheduling As traffic volume and complexity increase, resource-based scheduling can improve the efficiency, flexibility, and control of your Web site traffic. To get these benefits, you must create scheduling rules that direct different kinds of requests to different servers. You also specify a loadbalancing policy to determine which of several eligible servers handles a given request. You can specify port mappings to send client requests for any of a set of ports to a specific server port. And, you can specify port load balancing to send a client request to any one of a set of identical processes listening on different physical ports on a particular server. Scheduling rules Scheduling rules match client requests with particular servers. This permits you to distribute Web content across servers, replicating content only to ensure high availability or to spread the load of serving heavily accessed data. When resource-based scheduling is in effect, Central Dispatch parses incoming requests. At a Central Dispatch site, a request arrives as a URL, which has the following format: service://host:port/pathname 1-20 Resonate Central Dispatch User Guide

31 Introducing Central Dispatch For example: service host port pathname The port specification is usually omitted when standard ports are used, so a typical URL looks like this: To match the values in the URL, a scheduling rule contains the following components: Service VIP (after a DNS lookup of the host) Port A path expression that Central Dispatch uses to perform a pattern match against the actual pathname specified in the URL (HTTP requests only) A scheduling rule also contains the names of one or more server nodes. These nodes are eligible to respond when a URL matches the service, VIP, port, and pathname elements of the scheduling rule. If only one server is eligible to respond, the scheduler node transfers the request to that server. If multiple servers are eligible, Central Dispatch uses the load-balancing policy to determine the best server to respond. Load-balancing policies When using resource-based scheduling, if multiple servers are eligible to respond to a request, the scheduler node uses a load-balancing policy to determine the best server to respond. You can choose from the following load-balancing options: Resonate Central Dispatch User Guide 1-21

32 Introducing Central Dispatch Default load-balancing is tuned to work well in most situations and for most Web sites. The policy considers a combination of the current CPU load and the number of open connections. The range for CPU load is from 0 (idle) to 100 (busy). Open connections generally number from 10 or fewer for less busy sites to around 1000 for busy sites. This policy gives more weight to the number of open connections, which gives a good prediction of the load and power of a machine in most cases. The result is good distribution of traffic to all nodes. Default load-balancing is not available for affiliated server nodes. CPU-based round-robin load balancing is an alternate loadbalancing policy also based on current CPU load and the number of open connections. In this policy, the CPU load gets more weight, which generally schedules more hits to the more powerful machines, displays a smoother traffic distribution, and handles cases where the number of open connections does not reflect a machine s power. CPU-based round-robin load balancing is not available for affiliated server nodes. Round-robin load balancing selects a server by picking the next server in the list of eligible servers. This option is available for affiliated server nodes. Weighted round-robin load balancing selects a server by distributing requests to servers in a manner proportional to the server weights assigned to those servers. This option is available for affiliated server nodes. Custom load balancing selects a server based on criteria that you specify explicitly, including CPU load, number of open connections, and network latency. Port mapping By default, when you define a scheduling rule, the port on the server that the connection is directed to is the same port that the client specifies in its request (or the default port for the protocol being used if the client specifies no port). However, you can choose to define your scheduling rules to take advantage of port mapping so that requests to a given port or any one of a set of ports (the virtual port) are 1-22 Resonate Central Dispatch User Guide

33 Introducing Central Dispatch automatically mapped to a specified port on the server nodes (the server port). Port mapping is useful in supporting legacy port numbers, such as at a site in which a well-known port number has changed but support of the old port number along with the new is desired. Also, port mapping allows the successful scheduling of traffic to a server even when a particular application process has failed on that server. Dynamic port mapping Sometimes you might want to run two different Web servers with different features or performance characteristics on the same server node. For example, Web server A might be very good at serving up static HTML pages, while Web server B might be better at running CGI scripts. In this scenario, you would first configure the two Web server processes, Process A and Process B, on each of the server nodes you want to schedule to so that each process is listening on a different port. You can then use port mapping in your resource-based scheduling rules to direct connection requests for static HTML pages to Process A s port and connection requests for CGI scripts to Process B s port. Port load balancing You can also configure Central Dispatch to direct client requests for a port to any one of a set of identical processes listening on different ports on a particular server. This capability is known as port load balancing. Port load balancing provides increased concurrency for a particular service on a server machine. For example, port load balancing may improve the performance of legacy, single-threaded applications running on multi-processing servers. You might also use port load balancing to improve availability by running multiple copies of the same server software on your server machines. During port load balancing, Central Dispatch has already determined which server should get the request. Central Dispatch applies a round- Resonate Central Dispatch User Guide 1-23

34 Introducing Central Dispatch robin algorithm to determine which port to choose on the server selected by applying the scheduling rules. If the connection to the selected port fails, the scheduler applies the rules again to select a server for that request, rather than trying the next available port on the original server. In port load balancing, the most-recently-used port is tracked by rule and not by port. This means that if there is more than one rule which specifies the same port, that port may receive multiple requests at the same time because those requests were directed to the same port by applying different scheduling rules to those requests. Combining port mapping and port load balancing Port mapping and port load balancing can be applied simultaneously to the same client request. When combined, port mapping and port load balancing support legacy port numbers and provide the benefits of concurrency. Scheduling summary Figure 1-7 summarizes the scheduler s sequence of operations. The scheduling operation compares the URL with a set of scheduling rules, resulting in a set of eligible servers. If the set of eligible servers contains more than one server, the load-balancing policy selects the best server to handle the request Resonate Central Dispatch User Guide

35 Introducing Central Dispatch URL Rules Scheduling Operation Eligible Servers Load-Balancing Operation Best Server for Request Figure 1-7 Operational sequence in Resource-based Scheduling Content distribution strategies Resource-based scheduling lets you distribute the resources at a Central Dispatch site in many useful ways. This section describes just two of them: server-based distribution and indexing distribution. Server-based distribution One way to implement a web site is to segregate content across servers based upon the kind of content to be accessed and the server software used to access that content. This method is called server-based distribution. For example, a commerce site might have an HTML-based catalog, graphics files that illustrate products, an order entry form processed by Resonate Central Dispatch User Guide 1-25

36 Introducing Central Dispatch CGI scripts, and a secure financial database behind a firewall as shown in Figure 1-8. Firewall Nodes Graphics Files CGI HTML Secure DB Figure 1-8 Server-based (functional) distribution The Central Dispatch scheduling rules for this site match client HTTP requests to servers as shown in Table 1-1. Client URL in HTTP Request Server Selected Pathname expression contains *.gif or *.jpg Node 1 or 2 Pathname expression contains /cgi-bin/ Node 3 Pathname expression contains *.html Node 4 Hostname and port match a VIP and port defined in an SSL scheduling rule Node 5 Table 1-1 Rules for server selection using server-based distribution 1-26 Resonate Central Dispatch User Guide

37 Introducing Central Dispatch Indexed distribution Another way to distribute Web resources is by subject indexing. Different server nodes at the site serve up different categories of content based on the pathname directory structure in the client request. This method is called indexed distribution. Using Central Dispatch with the indexed distribution method allows you to optimize the performance of high-traffic sites with divided, hierarchical structures. For example, a site might offer information on the arts, business, or travel, among other subjects. Figure 1-9 illustrates an indexed site. This site also replicates the home page on all of the nodes because it is requested so often Figure 1-9 /business /arts /travel Indexing distribution Home Page The Central Dispatch scheduling rules for this site match client URLs to servers nodes as shown in Table 1-2. URL Content Pathname contains file index.html Server Selected Any Pathname contains directory /business Node 1 or 2 Pathname contains directory /arts Node 3 or 4 Pathname contains directory /travel Node 5 or 6 Table 1-2 Server selection based on indexing distribution Resonate Central Dispatch User Guide 1-27

38 Introducing Central Dispatch At this site multiple server nodes for each subject (two nodes per subject in Figure 1-9) provide system-level redundancy and load balancing. Because all servers share the demand for the most frequently requested resource (the home page), users receive the home page quickly. The indexed approach makes it easy to distribute and redistribute content by moving directory structures. Restricting a server s requests to a small subset of total content also enhances its ability to use the disk and memory cache. This often reduces I/O overhead and increases overall system performance. Load Balancing Thresholds Each server node or port on a server node can have a maximum number of connections specified. When the number of connections reaches that threshold, the scheduler directs no more traffic to that node or port. New requests can go to failover servers as specified in Server failover on page 1-9. See Set up COS groups and thresholds on page 2-10 for how to set thresholds. Class of service thresholds Class of service (COS) thresholds allow an individual node in a site to give different levels of service to traffic directed by different rules. Traffic directed to a node from one rule could get a more open class of service (more connections allowed) than traffic directed to the same node from another rule. Each node can have three COS groups. For each group, you can set the total number of open connections allowed for the node and the number of open connections allowed to each port on the node. To set COS group values for a node, see Set up COS groups and thresholds on page Resonate Central Dispatch User Guide

39 Introducing Central Dispatch Every scheduling rule calls out a COS group for the node targeted by that rule. If the COS group called out by the rule allows unlimited connections (as the group is defined on the node), then the rule can send unlimited traffic to that node. If the COS group limits connections, then the rule cannot send traffic to the node after hitting one of the limits. See Specifying scheduling rules on page 4-24 for setting a rule s COS group. COS is set up at the node and does not depend on the VIP through which a request arrives. The class of service for any request that arrives at a node is determined by the COS group called out by the rule that directed the request. Protection from attacks Central Dispatch features allow you to protect your site from common forms of attack: denial of service and bandwidth allocation denial. Denial of service attacks Denial of service attacks use the SYN-Bomb approach of creating multiple TCP initial synchronizing packets to tie up memory. Soon, all memory is taken up with uncompleted connections. Central Dispatch now monitors the amount of memory held waiting for TCP connections to complete. When such memory usage reaches a preset soft limit, Central Dispatch uses a much shorter timeout for dropping uncompleted connections. If the shorter timeout does not stop even more memory being taken, and memory usage reaches a preset hard limit, Central Dispatch chooses a pending connection to replace for each new connection. You can tune this feature by setting the memory usage soft limit and hard limit for various services. The services for which you can set limits are: HTTP, FTP, IP Persistence, SSL, TELNET, and BTREE. Resonate Central Dispatch User Guide 1-29

40 Introducing Central Dispatch Bandwidth allocation denial A bandwidth allocation denial (BAD) attack is usually multiple requests for very large files that saturate the network. To deal with BAD attacks, Central Disaptch now monitors the amount of data received by a client through a single connection. If that amount goes over your chosen setting, Central Dispatch lowers the packet size for all further data sent through that connection. Man Pages and Help Files Central Dispatch has man pages for UNIX systems and text-based help files for Windows NT and Windows 2000 systems. The commands are: CDAction, dispatch-manager, loadg, pagealarm, rcmp, rxpstat, and tcpecho. To install the man pages on a UNIX system: Under your Resonate base installation directory, create one of the following directories (depending on your operating system): $RESONATE/bin/man/man1m (Solaris, HP-UX, and Linux) $RESONATE/bin/man/man1 (AIX) Be sure to set the permissions appropriately to both the man and man1m (or man1) directories so that they can be read and searched. Find the directory unix/man/man1m on the Central Dispatch/Commander CD-ROM. Copy all the files ending in.1m to the new directory. Make sure the files have (or are given) read permissions. For every user who needs to access the man pages, add the path listed below to their MANPATH environment variable. (If necessary, define and export this environment variable in the appropriate login or shell startup scripts.) $RESONATE/bin/man 1-30 Resonate Central Dispatch User Guide

41 Introducing Central Dispatch On Linux, modify the /etc/man.config file to include section 1m in the list of sections defined for the MANSECT keyword, as shown below: MANSECT 1:1m:8:2:3:4:5:6:7:9:tcl:n:l:p:o To use the text-based help files on a Windows NT or Windows 2000 system, look on the Resonate Central Dispatch Resonate Commander CD-ROM and find the directory D:\winnt\help or D:\win2K\help (assuming your CD-ROM drive is mapped to the letter D, for example). From the CD-ROM, copy all the files ending in.txt to any directory you like. Open the file for a given command to read about that command. (You can open the file in Notepad, Wordpad, or any other text editor, or you can use the type command from a command prompt.) Summary Central Dispatch allows you to flexibly and elegantly manage the servers in a Web site. The benefits of Central Dispatch include: Superior control over Web site traffic Resource-based Scheduling permits fine-grained control over the assignment of servers to meet user requests. With Resource-based Scheduling directing each request to the appropriate server, you can distribute content to optimize the use of hardware and software resources. High availability Central Dispatch provides failover among server nodes that have replicated data. With a backup scheduler node, Central Dispatch also provides scheduler failover to ensure site availability. Scalability A Central Dispatch site can easily grow or shrink as needs change. You can add, remove, or reconfigure server nodes without shutting down the site. Resonate Central Dispatch User Guide 1-31

42 Introducing Central Dispatch Performance Central Dispatch has a negligible effect on server performance. As server nodes are added to the site, Central Dispatch achieves near linear performance improvements. Low cost deployment A Central Dispatch site comprising standard workstations can achieve the performance of a more expensive machine. No dedicated loadbalancing computer hardware is required. Ease of management Dispatch Manager offers real-time feedback about site activity so that you can monitor the site and quickly respond to problems or changes in demand. When an error occurs, Central Dispatch can notify you by or pager. Content distribution options Central Dispatch enables great flexibility in the distribution of content across multiple servers. You do not need to replicate content or software licenses on every server. Class of Service thresholds A Central Dispatch node can give different levels of service to requests using the same scheduling rule. Flexible site architecture A Central Dispatch site can be dispersed across any IP network, traversing LANs, WANs, and many firewalls Resonate Central Dispatch User Guide

43 CHAPTER 2 Using Central Dispatch This chapter describes how to configure, start, and use a Central Dispatch site once you have installed the software on all the nodes in your site. (Read the Resonate Central Dispatch Resonate Commander Site Preparation Guide to learn how to plan and implement the installation of Central Dispatch. The Resonate Central Dispatch Resonate Commander Installation Guide for each supported platform describe how to install Central Dispatch.) This chapter assumes that all content is mirrored on every server in your Central Dispatch site. To learn how to manage a site where different servers have differing content, read Chapter 4, Using Scheduling Rules. Using Dispatch Manager Dispatch Manager provides two modes for managing your Central Dispatch Sites. In administration mode you can use Dispatch Manager to: Configure the site: add and delete nodes, assign responsibility for particular VIPs to particular schedulers, and create scheduling rules. Start and stop Central Dispatch, and apply or discard configuration changes. Reschedule failed servers. Display real-time statistics such as: hits per second, number of open connections, CPU load, and network latency. Resonate Central Dispatch User Guide 2-1

44 Using Central Dispatch Set up alarms that send or pager messages to the site administrator (or anyone else) when specified types of events occur. View events and manage messages. Monitor the status of the Web site by testing accessibility, node services, and VIP functionality. Configure Commander. See the Resonate Commander User Guide for details. In monitor mode you can use Dispatch Manager to view site behavior. However, you cannot reconfigure, stop, or start the site in monitor mode. Starting Dispatch Manager How you start Dispatch Manager depends on the platform you run it on: UNIX Update your search path by adding the RESONATE/bin directory to your PATH environment variable.then, at your shell prompt, enter the command dispatch-manager. Windows NT and Windows 2000 Click the Start button, then click Programs, Resonate Central Dispatch, Dispatch Manager. Dispatch Manager opens the Welcome screen as shown in Figure 2-1. Once a site is configured, you can enter a node name in the Connect to Node field to administer or monitor the site. In this case, you are prompted for a password after clicking OK. To administer the site, enter the administration mode account password; to monitor the site, enter the monitor mode account password. (For more information about these accounts, read the Resonate Central Dispatch Resonate Commander Site Preparation Guide.) 2-2 Resonate Central Dispatch User Guide

45 Using Central Dispatch Figure 2-1 Dispatch Manager welcome screen Note If this is the first time you are configuring a site, leave the Connect to Node field empty, click OK, then click Site Setup to configure the site. Views Dispatch Manager organizes actions on and information about your site into four main views: Site View allows you to monitor site scheduling statistics and view node status and statistics (see Figure 2-2). Resonate Central Dispatch User Guide 2-3

46 Using Central Dispatch Site Setup allows you to setup your Central Dispatch site, through a series of tabbed dialogs. Scheduling Rules allows you to create or modify scheduling rules (see the definition on page 4-1). Commander lets you configure and administer Dispatch Manager Commander. (To learn how, read the Resonate Commander User Guide.) Messages provides both detailed and at-a-glance views of the message log. Figure 2-2 Site View before configuring the site Use the buttons on the bottom of the window to select a view. 2-4 Resonate Central Dispatch User Guide

47 Using Central Dispatch Note If you have not licensed Commander, the Commander button is greyed-out and inactive. Setting up your site To configure a Central Dispatch site, begin by clicking the Site Setup button. Dispatch Manager displays a set of tabbed dialogs you use to set up your site. Begin with the System Nodes dialog (see Figure 2-3). Figure 2-3 System Nodes dialog of the Site Setup view Resonate Central Dispatch User Guide 2-5

48 Using Central Dispatch Use this tab to: Add a node to the site. Give the node an alias. Assign a server weight to a node for load-balancing (optional). Enable or disable a node as a server. Mark a node as an affiliated server. Configure a node to be automatically enabled as a server after a failure recovery. Set up thresholds and Class Of Service (COS) levels (optional). Delete a node from the site. Add a node to the site 1. Enter a hostname or IP address in the Node Name field. 2. Click Add. Dispatch Manager adds the hostname to the list on the left (the hostname list box). It looks up the host in the name service and displays its IP address. Note Specify the host using its IP address whenever possible. When a hostname has multiple IP addresses or is resolved to multiple IP addresses, always use an IP address in the Node Name field. Use the alias feature (below) to make the hostname appear in the Site View for easy identification. 3. Continue adding hostnames as necessary. Give a node an alias Enter an alias name in the Alias field. When viewing nodes on the Site View bar graph, each bar for a node is sized to display the node name. Long node names can cause the display to be wider than the available space. Using aliases that are shorter than the actual node names allows 2-6 Resonate Central Dispatch User Guide

49 Using Central Dispatch the display of more node bars at once. Also, if the Node Name was entered as an IP address, the alias makes for easier identification of the node. Assign server weight Dispatch Manager makes load-balancing decisions based on the scheduling rules you have assigned (or other criteria) and server weights.these weights are used to establish a preference for one server over another independent of other scheduling factors. When you install Central Dispatch all the nodes in the site are initially assigned the same server weight of 100. You can alter this by changing the weights assigned to the nodes. If different nodes have different server weights, nodes with larger weights are preferred for scheduling over nodes with smaller weights, all other factors being equal. Note The default load balancing used by Central Dispatch works well if you are using one scheduler. If you are using multiple schedulers, Dispatch Manager recommends changing the servers weights to 50. After observing your site under load, you can adjust the servers weights to fine-tune performance. To assign a new server weight to a node: 1. Enter a new name in the Node Name text field. OR Select a hostname from the hostname list box. 2. Enter a number value in the Server Weight field (between 1and 100) or accept the default (100). Note When modifying the Server Weight field, you must press Return to allow changes to take effect. Resonate Central Dispatch User Guide 2-7

50 Using Central Dispatch Enable a node as a server 1. Enter a new name in the Node Name text field. OR Select a hostname from the hostname list box. 2. Click Server Enable. This option is checked by default. Note You may not want to enable all of the nodes in your site as servers. For example, you create a dedicated scheduler node by disabling the node as a server and then enabling it as a scheduler. You may also want to disable a server node for a time to perform maintenance on it. Automatically re-enable a recovered node as a server When a server fails, then recovers, you can manually re-enable the node as a server. You can also have Central Dispatch automatically reenable the server after each failure recovery. 1. Enter a new name in the Node Name text field. OR Select a hostname from the hostname list box. 2. Click Server Auto re-enable. This option is checked by default. If you do not check this option, you will need to manually enable the node as a server, after each failure recovery. 3. Click on Apply Changes. Mark a node as an affiliated node An affiliated node is a server-only node that does not need Resonate software installed. To mark a node as an affiliated node, click Affiliated Server. A node that does have Resonate software installed can be switched from standard to affiliated using this feature. There is no need to stop 2-8 Resonate Central Dispatch User Guide

51 Using Central Dispatch the site. Note that with Windows NT and Windows 2000 machines, you can switch the machine from standard to affiliated in the following ways: Set the newly affiliated server to communicate with full NAT. (Done with the Routing mechanism for Affiliated Servers option, Advanced Options button, Operations tab, Site Setup view in Dispatch Manager. Set the newly affiliated server to communicate with half NAT and reboot the server. Set the newly affiliated server to communicate with half NAT and use a new VIP for that server. For UNIX and Windows systems, on each affiliated server using half NAT, be sure that routing is set up so that traffic goes back through the Central Dispatch scheduler. If you change a VIP for an affiliated server that is using half NAT, be sure that on the server, you remove the static route for the old VIP and add a static route for the new VIP. The static route should send all replies to clients back through the scheduler. With half NAT and affiliated servers, a response to a client does not automatically go back through the scheduler. The routing table entry for a VIP must be set up to send responses back through the scheduler. Also, the affiliated server must be on the same subnet as the scheduler. A benefit of half NAT is that the affiliated server sees a client s real IP address. With full NAT and affiliated servers, the server automatically sends responses back through the scheduler. You do not need to make changes to the server s routing table and affiliated servers do not have to be on the same subnet as the scheduler. With full NAT, the affiliated server does not see a client s IP address. Resonate Central Dispatch User Guide 2-9

52 Using Central Dispatch Note A site with affiliated servers cannot have a scheduler using a Central Dispatch software from any version earlier than version 3.1. Schedulers must be version 3.1 or later to recognize affiliated servers. Set up COS groups and thresholds COS (class of service) groups limit the number of connections allowed to a node or to the ports on a node. Every scheduling rule calls out a COS group of A, B, or C to give a COS value to traffic directed by the rule (see Specifying scheduling rules on page 4-24). A rule can select a server only if the selection would not violate either of the COS thresholds (server or port). The value of Unlimited is the default and means that the server or port for that COS group has no threshold. An entry of -1 gives the same value as unlimited. By using only one COS group, each server node or port on a server node can have a maximum number of connections specified, giving a load balancing threshold. When the number of connections reaches that threshold, the scheduler directs no more traffic to that node or port. New requests can go to failover servers. Note Every Central Dispatch scheduler keeps its own COS counts. If two schedulers access the same server, and each scheduler has a COS limit of 20 connections for that server, they will attempt to open a total of 40 connections. Figure 2-4 shows the COS set up box. For each group you can enter the maximum number of server and port connections allowed. Note when choosing values that: The number of server connections must be greater than or equal to the number of port connections Resonate Central Dispatch User Guide

53 Using Central Dispatch Figure 2-4 COS open connections set up box The number for port connections is for each port on the server. For example, a value of 100 for server connections and 20 for port connections means that all the ports on that server are limited to 20 connections each. All the port connections together are limited to 100. You could have 5 ports with 20 connections each, 10 ports with 10 connections each, or any other mix, as long as an individual port does not exceed 20 and all ports together do not exceed Select a node name from the node list box. 2. Enter the maximum number of connections allowed for the Server and Every Port on the Server for each group. 3. Click Add. Delete a node from the site 1. Select the hostname from the list. 2. Click Delete. Note You cannot delete a node that has been designated as a primary scheduler or backup scheduler from your Central Dispatch site. To delete a scheduler node from the site, first make it a server node, then delete it. Resonate Central Dispatch User Guide 2-11

54 Using Central Dispatch Creating VIPs To create a VIP, specify a hostname or an IP address. Any name or IP address used for a VIP must be registered with DNS, but must not be assigned to a computer. Using a VIP with an IP address that conflicts with another IP address on the network causes connectivity problems with nodes and possibly with the site. A VIP with an conflicting IP address is the same as using an IP address in two places on the same subnetwork. Add a VIP 1. Click the VIPs tab to open the VIPs dialog (see Figure 2-5). Figure 2-5 VIPs dialog of the Site Setup view 2. Enter the site IP address in the VIP field Resonate Central Dispatch User Guide

55 Using Central Dispatch 3. Click Add. Dispatch Manager adds the VIP to the list on the left. If you enter a name, Dispatch Manager looks up its VIP (using DNS) and displays it in the IP Address text box. 4. Designate the primary and backup schedulers for the VIP you just entered. To learn how, read Designating schedulers on page Set the Use Shadow Scheduling checkbox. See Shadow Scheduling on page 2-20 to learn about shadowing. Continue adding to the list until it contains all of the VIPs the site will serve. Delete a VIP 1. Select the VIP from the list box. 2. Click Delete. Designating schedulers Any node (except an affiliated server node) in the Central Dispatch site can be a scheduler (including nodes that are enabled as servers). If you are using one scheduler at your site, the machine you configure as the scheduler should be powerful enough to handle all the incoming traffic to your site. Otherwise, the overall performance of your site will suffer if the scheduler bogs down. You can use multiple schedulers in your site to achieve high performance levels. When configuring multiple schedulers, keep throughput and CPU load below maximum levels. If you assign a primary scheduler as a backup scheduler for another primary scheduler, be sure that the total load to the scheduler is below maximum levels. After observing your site under load, you can adjust the scheduler/server weights to fine-tune performance. Each scheduler license includes a backup scheduler. You can designate the backup scheduler to automatically take over when the primary scheduler is unavailable. When the primary scheduler comes back on- Resonate Central Dispatch User Guide 2-13

56 Using Central Dispatch line, it automatically reintegrates into the site and again assumes the primary scheduler role. Note To recognize affiliated servers, a scheduler must be on either a Solaris, HP-UX, Windows NT, or Windows 2000 platform. A scheduler on a Linux or AIX platform cannot recognize affiliated servers. Specify primary and backup schedulers 1. Display the VIPs tab (see Figure 2-5). 2. Select the VIP whose scheduler(s) you wish to designate. The name of the VIP and its IP address are displayed in VIP and IP Address. 3. Pull down the Primary Scheduler list. Figure 2-6 Policies dialog of the Site Setup view 2-14 Resonate Central Dispatch User Guide

57 Using Central Dispatch The list of the nodes currently in the site is displayed. 4. Click the name of the node you want as the primary scheduler. 5. If you want, pull down the Backup Scheduler list and select a different node as your backup scheduler. Caution The primary and the backup schedulers must be different computers. However, they must both be located on the same subnet. Choosing scheduling policies When the scheduler receives an incoming request, it determines which node or set of nodes is qualified to respond to the request. You can define either a round-robin or a resource-based scheduling policy (see Figure 2-6). Round-robin scheduling policy If you select round-robin as your scheduling policy, the scheduler passes a client request to the next available server node in the site. You should use this policy only in a fully-replicated site one where all servers have the same content. To specify round-robin, choose Round-robin from the Scheduling Policy pull-down menu. If you choose round-robin, you do not need to specify scheduling rules or a load-balancing policy. In fact, any rules you specify will be ignored. However, if you later switch to the Resource-based scheduling policy, all the scheduling rules you have specified go into effect. Resonate Central Dispatch User Guide 2-15

58 Using Central Dispatch Note The round-robin scheduling policy is very different from the roundrobin load-balancing policy. To learn more about the round-robin load-balancing policy, read Round-robin (basic) on page Resource-based scheduling policy Resource-based scheduling lets you distribute content to different nodes instead of mirroring all the content on all of the nodes. When you select Resource-based scheduling, the scheduler determines which servers are eligible to respond to a client request by matching the syntax of the request (such as a URL) against scheduling rules. When you first install Central Dispatch, the scheduler uses default scheduling rules. These rules make all servers eligible for all requests. You can customize the scheduling rules to reflect the way you have distributed content at your site. (For more information about scheduling rules, read Chapter 4, Using Scheduling Rules.) If, according to the scheduling rules, a particular client request can be serviced by more than one server, the scheduler uses the current loadbalancing policy to determine which server (among those eligible) should handle the request. (To learn about load-balancing policies, read Choosing a load-balancing policy on page 2-17.) To enable Resource-based scheduling, choose Resource-based from the Scheduling Policy pull-down menu. When you select Resource-based scheduling, the Resource-based Load Balancing Policy pull-down menu and the Scheduling Rules button (at the bottom of the screen) become active: 1. Choose a load-balancing policy. To learn how, read Choosing a load-balancing policy on page Create scheduling rules. To learn how, read Chapter 4, Using Scheduling Rules Resonate Central Dispatch User Guide

59 Using Central Dispatch Choosing a load-balancing policy If you select Resource-based as your scheduling policy, you must also select a load-balancing policy. (If you select Round-robin as your scheduling policy, no load balancing is done, so no load-balancing policy can be selected.) The load-balancing policy you select is used when more than one server is eligible to respond to a client s request. To specify a load-balancing policy, click the Site Setup button, then click the Policies tab. Click the Resource-based Load balancing Policy drop-down list and select one of the following: CPU Load and Connections (Basic) CPU Load and Connections (Enhanced) Round Robin (Basic) Round Robin (Enhanced) Custom Note Affiliated servers can use only the round-robin and round-robin (enhanced) policies. CPU Load and Connections (Basic) CPU Load and Connections (Basic) load-balancing uses a predefined policy that works well in most cases. The policy considers a combination of the current load and the number of open connections. This policy is tuned to work well with various network conditions, heterogeneous computers, and varying loads. The range for CPU load is from 0 to 100 (idle to busy). Open connections generally number from 10 or fewer for less busy sites to around 1000 for busy sites. This policy gives more weight to the number of open connections, which gives a good prediction of the load and power of a machine in most Resonate Central Dispatch User Guide 2-17

60 Using Central Dispatch cases. The result is good distribution of traffic to all nodes. (Not available for affiliated servers.) CPU Load and Connections (Enhanced) CPU Load and Connections (enhanced) load balancing is an alternate policy that, like the CPU Load and Connection (basic) policy, is based on current CPU load and the number of open connections. In this policy, the CPU load gets more weight, which generally schedules more hits to the more powerful machines, displays a smoother traffic distribution, and handles cases where the number of open connections does not reflect a machine s power. (Not available for affiliated servers.) Round-robin (basic) Round-robin load-balancing rotates assignments among the eligible servers for a particular scheduling rule. When an incoming URL (or other client request) matches a scheduling rule, the scheduler assigns a server from among those eligible to respond to that rule, choosing a different server each time it uses the rule. Round Robin (Enhanced) Round robin (enhanced) load balancing selects a server by distributing requests to servers in a manner proportional to the server weights assigned to those servers. Note The round-robin load-balancing policy is very different from the round-robin scheduling policy. To learn more about the round-robin scheduling policy, read Choosing scheduling policies on page Resonate Central Dispatch User Guide

61 Using Central Dispatch Custom load-balancing Custom load-balancing lets you specify alternative primary and secondary factors to apply in load-balancing decisions. (Not available for affiliated servers.) When you choose Custom from the Resource-based Load Balancing Policy pop-up menu, the options under Custom Load-balancing Priorities become available (see Figure 2-7). Figure 2-7 Selecting custom load-balancing policies The custom load-balancing options you should select are: CPU load Number of open connections. (This is the number of open connections through the VIPs.) Resonate Central Dispatch User Guide 2-19

62 Using Central Dispatch Shadow Scheduling Choose either of the options as the primary factor and the other as the secondary factor. With custom load-balancing, the primary factor influences scheduling decisions the most. Setting up the CPU as primary criteria is useful when machine power is not well described by open connections. An example is a server handling CPU intensive CGI scripts and serving HTML pages at the same time. Scheduling based on open connections could lead to poor load-balancing as a server with open connections may not be able to handle CPU intensive requests. Using CPU load as the primary criteria may not be as smooth as giving more weight to the number of open connections. Setting up the number of open connections as primary is useful at less busy sites where the number of open connections is a good indicator of a machine s power. The faster the machine, the faster it closes an open connection. An example is a site with traffic of either pure HTML or CGI and a fast line. This policy also quickly handles web server hangs or anomalies. The network latency option in custom load-balancing measures the network latency between schedulers and servers. Specifically, it measures in tens of microseconds the time it takes for a SYN packet to travel from a scheduler to a server plus the time it takes for the SYN ACK packet to travel from the server back to the scheduler. Network latency measurements are kept for each scheduler, but are not displayed by scheduler in the Site View. The Site View displays the largest network latency value (from all the schedulers) for a server. The site scheduler keeps a persistent session table to keep track of persistent session connections. If the site has a primary scheduler and a backup scheduler, you can use the shadow scheduling feature to save the persistent session table if the primary scheduler fails. You set shadow scheduling for each VIP in the site. To set the feature in Dispatch Manager, bring up the Site Setup view. In the VIP tab, 2-20 Resonate Central Dispatch User Guide

63 Using Central Dispatch select the target VIP and then click the Use Shadow Scheduling checkbox to turn shadow scheduling on or off. Shadow scheduling synchronizes the persistent sessions table from the primary scheduler to the backup scheduler. The synchronization happens at two levels. Session creation. Whenever a persistent session is created, the primary scheduler sends the session information to the backup scheduler using the kernel. This action is instantaneous but unreliable. Table synchronization. At an interval set through the Dispatch Manager, the complete persistent sessions table is synchronized from the primary scheduler to the backup scheduler using agents. This synchronization method is reliable. Note Session information from the kernel is sent as an IP message with no guaranteed service, meaning that individual messages may be lost. Table synchronization using agents reliably updates any messages lost since the last synchronization. To set the table synchronization interval, start Dispatch Manager and bring up the Site Setup view. From the Operations tab, click on the Advanced Options button. Enter a value (of minutes) in the Shadow Scheduling Synchronization Interval field. Click OK in the Advanced Options box, and Apply Changes on the Operations tab. Also in the Advanced Options box, consider the Enable Primary Scheduler Failback checkbox. As the number of persistence table entries grows, disabling failback is a highly recommended strategy. With failback, the complete persistence table is updated at once, which can cause noticeable disruptions in service. The Enable Primary Scheduler Failback checkbox determines what happens when a primary scheduler fails, a backup scheduler takes over, and then the primary returns to life. With the box checked, the Resonate Central Dispatch User Guide 2-21

64 Using Central Dispatch primary scheduler does take over as the active scheduler when the primary scheduler returns to life. With the box blank, the backup scheduler remains as the active scheduler even after the primary scheduler returns to service. With this option, if you want to once again have the primary scheduler be the active scheduler, you can make the switch manually using the Failback to Primary Scheduler button in the Start Site box. Note To make any changes using the Advanced Options window, the Central Dispatch site must be stopped. The persistent session table can get very large. When a table synchronization takes place, the default block size for data transfer is 100,000 bytes. If you want to tune the block size for faster transfer on a UNIX system: 1. Use the cdagentctl command to stop Central Dispatch agents. 2. Edit the file $RESONATE/bin/cdagentctl. 3. Add and export the following environment variable. Follow the format of existing variables in the file: RES_PERSIST_BLOCK_SIZE=<value> export RES_PERSIST_BLOCK_SIZE Where <value> is the size in bytes. 4. Use cdagentctl to restart the agents. To tune the block size for faster transfer on a Windows NT system: 1. From Start, open Settings, Control Panel, System. 2. Open the Environment tab. Add to the System Variables: RES_PERSIST_BLOCK_SIZE and your chosen value. To tune the block size for faster transfer on a Windows 2000 system: 2-22 Resonate Central Dispatch User Guide

65 Using Central Dispatch 1. From Start, open Settings, Control Panel, System. 2. Open the Advanced tab. 3. Open Environment Variables. 4. Add to the System Variables: RES_PERSIST_BLOCK_SIZE and your chosen value. Address changes and schedulers The MAC addresses for all the servers in a Central Dispatch site are cached in the RXP module on the schedulers. If you do anything to change the MAC address of any of the servers in the site (such as changing a NIC card on a Central Dispatch scheduler or changing a NIC card or modifying a route on a router used by Central Dispatch) the schedulers will not be able to transfer client connections to that server. When this happens, you will observe in Dispatch Manager that the number of hits per second on that server is zero. To correct this, you must force the schedulers to flush the cache and obtain the updated MAC addresses. To flush the cache: Using Dispatch Manager, delete the server with the new MAC address from the site, then add the server back into the site. Affiliated servers and master agents Central Dispatch master agents in any release before version 3.1 can not recognize affiliated servers as part of a Central Dispatch site. Master agents on Linux or AIX machines also cannot recognize affiliated servers. That means that a site with affiliated server nodes can have the site master agent on only Solaris, HP-UX, Windows NT, or Windows 2000 machines using version 3.1 or later. If your site has Linux or AIX nodes or is mixed with some nodes using version or 3.0 AND Resonate Central Dispatch User Guide 2-23

66 Using Central Dispatch Your site includes affiliated servers THEN Set the NOMASTER variable on all nodes using Linux, AIX, Central Dispatch version or 3.0. The variable stops agents on the node from being selected as a master agent. On UNIX systems: In the base Resonate bin directory, find the script cdagentctl. Using the default Resonate base directory as an example, the path is /usr/local/resonate/bin. Add the following two-line entry to the list of variables: RES_NOMASTER= export RES_NOMASTER On Windows NT systems: From the Start menu, select Settings, Control Panel, System, Environment. Set a System Variable of RES_NOMASTER. The variable does not need a value. Reboot the system. On Windows 2000 systems: From the Start menu, select Settings, Control Panel, System, Advanced, Environment Variables. Set a System Variable of RES_NOMASTER. The variable does not need a value. Reboot the system. Affiliated servers and scheduling Central Dispatch has two ways to schedule traffic to affiliated servers: half NAT and full NAT. To set the routing to half NAT or full NAT, go to the Site Setup view, Operations tab, Advanced Settings button Resonate Central Dispatch User Guide

67 Using Central Dispatch Half NAT Using half NAT, the scheduler accepts traffic on the VIP, forwards the traffic to the affiliated server, but keeps the source IP address the same. Responses from the affiliated server must go back to the scheduler so that the scheduler can change the source IP address to the VIP address and maintain the VIP s identity. For responses to go back through the scheduler, it must be the default router/gateway for the affiliated server. With half NAT: The routing table change on the affiliated server may create difficulties with direct access to the server. Affiliated servers must be on the same subnet as the scheduler. The affiliated server sees a client s real IP address. Full NAT Using full NAT, the scheduler accepts traffic on the VIP and forwards the traffic to the affiliated server by changing the source IP address to that of the scheduler. The affiliated server then automatically sends all responses back to the scheduler. The scheduler then changes the IP address of the response to that of the VIP to maintain the VIP s identity. With full NAT: Affiliated servers see all requests as coming from the scheduler. Client identities are lost to the affiliated server. No routing table change is needed on the affiliated server. Affiliated servers do not have to be on the same subnet as the scheduler. Resonate Central Dispatch User Guide 2-25

68 Using Central Dispatch Configuring groups of nodes Grouping nodes has two advantages. The first gives selective viewing with Site View. Choosing Single Group as a view option for the Site View bar graph display shows only those nodes in the chosen group. The Single Group option is found in the far right option box of the Site View options panel that is just above the bar graph display area. The second advantage is that under the Commander view, you can set up a rule once and apply it to multiple nodes. If you have 20 nodes that will all use the same rule, you do not have to manually apply the rule to each node. Simply group the nodes and apply the rule to the group. Figure 2-8 shows the Groups tab: Figure 2-8 Creating node groups 2-26 Resonate Central Dispatch User Guide

69 Using Central Dispatch The Groups field on the tab shows the names of all the groups defined for the site. The Members field shows which nodes are in a selected group. To add a group, click the Add button. The dialog box that appears with the following fields: Group takes the name for the new group. The group name can include alpha and numeric characters and the - (hyphen) and _ (underbar) characters. Other special characters are not allowed. Members lists the members of the group. This box is empty until you add members. Non Members is a list of all the nodes in the site. Use the arrows between the Members and Non Members boxes to move a node from one box to another. Click OK when you are done with that group. The group then shows on the Groups tab and selecting the group displays the group members. Setting configuration options Before starting the site, you may want to select or change configuration options available on the Operations tab. Set options 1. From Site Setup, click the Operations tab to open the Operations dialog (see Figure 2-9). From the Stop Site box, you can: Import a saved configuration file. Select advanced configuration options. Stop Central Dispatch. Note All operations in the Stop Site box require a stopped site. Resonate Central Dispatch User Guide 2-27

70 Using Central Dispatch Figure 2-9Operations dialog of the Site Setup view 2. If you want to set the advanced options, click the Advanced Options button (see Figure 2-10). In this dialog you can set the Agent Heartbeat Interval, the value of Heartbeats Until Down, the Shadow Scheduling Synchronization Interval (see Shadow Scheduling on page 2-20 for shadowing information), the Enable Primary Scheduler Failback option the Use Client Route Caching option, and the NAT option for affiliated servers. Agent Heartbeat Interval sets the time between agent communications (in seconds). You can adjust this interval by entering a new value in the Agent Heartbeat Interval text box. The heartbeat interval is set to depend upon the network latency between the nodes in your site. If the nodes are connected over a WAN link (which is slower than a LAN link) or you have a large site with many nodes (more than 10), you may want to use a 2-28 Resonate Central Dispatch User Guide

71 Using Central Dispatch Figure 2-10 Advanced Options dialog longer heartbeat interval. The heartbeat interval must be between 1 and 5, inclusive. The Heartbeats Until Down parameter defines how many heartbeats must go by without an agent responding before the server on which that agent is running is considered down. You can adjust this parameter in the Heartbeats Until Down text box. If the values for the heartbeat parameters are too small at a large or very busy site, the agent log file may grow rapidly. You can increase the values for both parameters to keep the log file size reasonable. The Shadow Scheduling Synchronization Interval determines how often the persistence sessions table is synchronized between the primary and backup Central Dispatch schedulers. The Enable Primary Scheduler Failback checkbox determines what happens when a primary scheduler fails, a backup scheduler takes over, and then the primary returns to life. With the box checked, the primary scheduler does take over as the active scheduler when the primary scheduler returns to life. With the box unchecked, the backup scheduler remains as the active scheduler even after the primary scheduler returns to service. This feature is particularly useful when using shadow scheduling, as disabling failback creates the least disruption to the persistence Resonate Central Dispatch User Guide 2-29

72 Using Central Dispatch sessions table. With this option off, if you want to once again have the primary scheduler be the active scheduler, you must make the switch manually using the Failback to Primary Scheduler button in the Start Site box. With this option off, if the whole site fails, as with a power outage, the Auto Site Restart feature may bring up the site with the backup scheduler as the active scheduler, depending on which machine recovers first. The Use Client Route Caching checkbox is also for affiliated servers. Client route caching uses past information to return packets to a client across the same route. This may improve performance in a CPU-bound system. The trade-off is that if a change in network topology changes the route to the client, responses are delayed until the new route is determined. The Routing mechanism for Affiliated Servers menu gives the choice of Half NAT or Full NAT for forwarding TCP/IP traffic. The default is half NAT. On each affiliated server using half NAT, be sure that routing is set up so that traffic goes back through the Central Dispatch scheduler. If you change a VIP for an affiliated server using half NAT, be sure that on the server, the static route for the VIP sends all replies to clients back through the scheduler. Using live operations The Operations tab Start Site box gives a number of options. Use live options From Site Setup, click the Operations tab to open the Operations dialog (see Figure 2-9). From the Start Site box, you can: Start Central Dispatch. Export a saved configuration file Resonate Central Dispatch User Guide

73 Using Central Dispatch Manually return the site to a primary scheduler after a failover to a backup scheduler. The Failback to Primary Scheduler button returns the role of active scheduler to a primary scheduler if the site is currently active using a backup scheduler. The option is available only if the Enable Primary Scheduler Failback checkbox in the Advanced Options button of the Stop Site box is not checked. Disconnect from a site. Note All operations in the Start Site box require a running site. Licensing the site Before starting a Central Dispatch or Commander site, you need to enter the license activation key. Look for the key on the Resonate Central Dispatch/Commander CD-ROM. Enter the key in the New Key field of the Licensing tab. A Central Dispatch site will not start unless the site has a valid key. The Days Until Expiration field shows a limited number if the key is a product evaluation key. The Number of Nodes and Number of Schedulers depend on the number of licenses bought for the site. Any changes in the products or number of products that you license require a new license key from Resonate. Your Resonate sales representative should order a new key for your site when needed. If you are upgrading a current Central Dispatch site to a new release, the current site license is fully functional for the new release. Simply use the current license when setting up the new release. An upgraded site needs a new license only when adding new licensed features to the site or when extending the length (in time) of the license. Resonate Central Dispatch User Guide 2-31

74 Using Central Dispatch Figure 2-11 Licensing dialog of the Site Setup view Starting the site When you start your site, Central Dispatch: Configures each node in the site as a server, a scheduler, or both, and configures its network interfaces for virtual IP addresses. Activates communication among Central Dispatch agents. Enables communication between Dispatch Manager and the site, so that Dispatch Manager can display site statistics. Configures the schedulers. Distributes license information to all nodes Resonate Central Dispatch User Guide

75 Using Central Dispatch In addition, Dispatch Manager connects to and starts the Commander hosts, if they have been configured. (To learn more about Commander, read the Resonate Commander User Guide.) Note Before starting a site, make sure that: - The license key is entered in the Licensing tab of the Site Setup view. - At least one node is set up in the System Nodes tab of the Site Setup view. - At least one VIP is set up in the VIPs tab of the Site Setup view. If the site does not start, check the message panel at the bottom of the Dispatch Manager for an error message. Start the site 1. From Site Setup, click the Operations tab to open the Operations dialog. 2. Click Start Site. 3. Enter the administration mode account password (see Figure 2-12). Figure 2-12Password dialog A message in the status bar window confirms that the Central Dispatch site is establishing itself. 4. Check the message log to make sure that no errors have occurred. To learn how, read Viewing messages on page Resonate Central Dispatch User Guide 2-33

76 Using Central Dispatch You have now configured and started the Central Dispatch site. The Central Dispatch icon in the lower right corner of the window starts pulsing, indicating that Dispatch Manager is communicating with the site. Also, if Commander is running, the Commander icon immediately to the left of the Central Dispatch icon also pulses. To shut down Central Dispatch read Stopping the site on page To exit Dispatch Manager, read Exiting Dispatch Manager on page Changing a running site When Dispatch Manager is connected to a running site, it displays the existing site configuration until you change the configuration. When you reconfigure any of your site parameters using Dispatch Manager, the Apply Changes and Discard Changes buttons become active. These buttons are available on each dialog that lets you change your configuration. When you click Apply Changes, Dispatch Manager sends the new configuration information to the site, where it takes effect immediately. When you click Discard Changes, Dispatch Manager reverts to the last applied configuration. Verifying your changes Changing the configuration of a running site can have unwanted consequences unless you do it carefully. Make one change at a time and verify the results before making another change. Make sure you are satisfied with the information displayed in all of the dialogs before applying the changes Resonate Central Dispatch User Guide

77 Using Central Dispatch Caution Applying changes in Dispatch Manager applies the current values of all dialogs, not just the one that is currently visible. Before removing or disabling nodes, check for possible consequences of the change. For example, if you use Resource-based Scheduling and you delete the only servers that can respond to certain kinds of requests based on your current scheduling rules, your site will not be able to satisfy those requests until you restore at least one of those nodes. You may want to replicate the data on those servers to other servers in the site (and adjust the scheduling rules accordingly) before removing or deleting those servers. Examining a server s scheduling rules To examine a server s scheduling rules: 1. Display the Site View tab of Dispatch Manager. 2. Click on the node in the bar graph to see the node view. 3. Look for any highlighted scheduling rules. A highlighted scheduling rule means that the node is the sole server for that rule. Note If you disable or remove a node, the scheduling rules that reference that node remain in place. You can therefore temporarily disable or remove a node without having to recreate the rules when you reinstate the node. If you permanently remove a node, however, you must manually update the rules. Resonate Central Dispatch User Guide 2-35

78 Using Central Dispatch Exporting and importing a configuration This section describes how to export (save) your current Central Dispatch site configuration and to import it after a shutdown. Exporting a configuration Whenever you send a configuration to the site by starting Central Dispatch or make configuration changes, you can save a copy of the configuration information into a file. Once a site is running, you do not need a configuration file because Central Dispatch agents maintain the site s configuration. Central Dispatch itself never accesses the configuration file. This file simply provides a backup in case you need to recover your configuration. Export a configuration 1. From Site Setup, click the Operations tab. 2. Click Export Configuration. 3. In the Export Configuration dialog box, specify the directory and filename where you want to save the configuration. 4. Click Save. This operation saves the configuration to a file on the computer where Dispatch Manager is running. Importing a configuration Before starting or restarting a site, you can import a previously saved configuration file. Import a configuration 1. Run Dispatch Manager on the system where the configuration file resides. 2. From Site Setup, click the Operations tab Resonate Central Dispatch User Guide

79 Using Central Dispatch 3. If the site is active, stop the site using the Stop Site button. 4. Click Import Configuration. 5. In the Import Configuration dialog box, specify the path of the configuration file. To restore a configuration file, specify the name of the saved file you want to restore. 6. Click OK to load the configuration into Dispatch Manager. 7. Start the site. Disabling or removing nodes After a site is operating, there may be times that you need to disable or remove a node. If you are using resource-based scheduling, consider the effect on users before disabling or removing a node. Examine the node s scheduling rules as follows: 1. Display the Site View tab of Dispatch Manager. 2. Click on the node in the bar graph to see the node view. 3. View the scheduling rules for the node. Look for highlighted rules. A highlighted rule indicates that the node is the sole server for the rule. Users whose requests match the rule will receive error messages if the node is unavailable. To avoid this problem, you can replicate the data on another server and revise the rule to send requests to both servers. Disabling a node prevents new connections to that node, but does not drop existing connections. You can first disable a server, wait for clients with existing connections to that server to close the connections, then remove the node from the site if that is your goal. Clients then see no disruption of service. Resonate Central Dispatch User Guide 2-37

80 Using Central Dispatch Note When you disable or remove a node, scheduling rules that reference the node remain in place. You can therefore temporarily disable or remove a node without having to recreate the rules when you reinstate the node. If you permanently remove a node, however, you must manually update the rules. Stopping the site When you stop your site, all connections that are still open are dropped. Before stopping the site, use the Site View tab of Dispatch Manager to check the number of open connections. Stopping the site also stops your Commander hosts. (For more information about Commander, read the Resonate Commander User Guide.) Stop the site 1. Click the Site Setup button, then click the Operations tab. 2. Click the Stop Site button. When you stop Central Dispatch, Dispatch Manager: Stops the scheduler from assigning requests to nodes. Stops current outgoing server-to-client transmissions. Stops communication among the Central Dispatch agents. Stops communication between itself and the site. Disconnects all other instances of Dispatch Manager connected to the site. Puts Commander hosts into the idle state. (For more information about Commander, read the Resonate Commander User Guide.) Shutting down the site does not stop Dispatch Manager. To exit Dispatch Manager after stopping the site, close the Dispatch Manager 2-38 Resonate Central Dispatch User Guide

81 Using Central Dispatch window by selecting Close from the window manager menu (UNIX) or the window menu (Windows NT and Windows 2000). Starting and stopping Central Dispatch Agents UNIX You can start and stop Central Dispatch agents from the command line on UNIX, Windows NT, or Windows 2000 or from the Services control panel on Windows NT and Windows You can stop an agent (provided you have root privileges) with the following command: Resonate installation directory/bin/cdagentctl stop You can start an agent by entering: Resonate installation directory/bin/cdagentctl start To stop and then start an agent with one command, enter: Resonate installation directory/bin/cdagentctl restart Windows NT and Windows 2000 On Windows NT and Windows 2000 a Central Dispatch agent is installed as a service that starts automatically when Windows NT or Windows 2000 restarts. You can stop an agent from a command prompt by entering: Resonate installation folder\bin\cdagent stop You can start an agent from the command line by entering: Resonate installation folder\bin\cdagent start Resonate Central Dispatch User Guide 2-39

82 Using Central Dispatch To use the Services control panel with Windows NT, click Start, then Settings to display the Control Panel. Double-click Services, then select Resonate Central Dispatch Agent in the Service list box. Click Stop to stop the service (if it is currently running); click Start to start the service (if it is currently stopped). When starting the agent, you can give a port number for the agent in the Setup Parameters field. The following is an example of a port entry in the Startup Parameters field: -p 2114 To use the Services control panel with Windows 2000, click Start, Programs, Administrative Tools, and Services. Select Resonate Central Dispatch Agent in the Service list box. Click Stop to stop the service (if it is currently running); click Start to start the service (if it is currently stopped).when starting the agent, you can give a port number for the agent in the Setup Parameters field. The following is an example of a port entry in the Startup Parameters field: -p 2114 Solaris UDP Buffers When starting a Central Dispatch agent, Central Dispatch sets a UDP maximum buffer size (the network attribute udp_max_buf). Central Dispatch does not change the value if that value is already larger than required for Central Dispatch. If Central Dispatch does change the value, a message is sent to the system console and to a log file. To activate the log file, edit the file /etc/syslog.conf and add (or uncomment if it already exists) the following line. You can use whatever path and file name is appropriate, but the path and file must already exist. daemon.notice /var/adm/messages 2-40 Resonate Central Dispatch User Guide

83 Using Central Dispatch AIX Socket Buffers When starting a Central Dispatch agent, Central Dispatch sets a maximum buffer size allowed for a socket (the network attribute sb_max). Central Dispatch does not change the value if that value is already larger than required for Central Dispatch. If Central Dispatch does change the value, a message is sent to the system console and to a log file. To activate the log file, edit the file /etc/syslog.conf and add (or uncomment if it already exists) the following line. You can use whatever path and file name is appropriate, but the path and file must already exist. daemon.notice /var/adm/ras/messages Removing a server from service Occasionally, you want to remove a server from a site for a reboot or replacement or to change the site configuration. The following procedures remove a server: The quickest and most abrupt way to remove a server is to go to the Site Setup view of the Dispatch Manager and display the System Nodes tab. Delete the server from the System Nodes list, and apply the change. This resets all user connections and forces them to other servers. A less disruptive option is to disable the server function on the target machine. To disable a server, go to the Site Setup view of the Dispatch Manager and display the System Nodes tab. Select the server from System Nodes, click on the Server Enable box to uncheck the box, and click the Apply Changes button. Once the server is disabled, no new TCP connections go to the server. New TCP connections from clients bound to the server by SSL or persistent scheduling rules also do not go to the server, but are redirected to other servers. Resonate Central Dispatch User Guide 2-41

84 Using Central Dispatch The last option takes more time, but is less disruptive when using SSL or persistent scheduling rules. Go to the Site Setup view of the Dispatch Manager and display the System Nodes tab. Select the server from the System Nodes list and set the Server Weight to one (1). This assumes that all other system weights are higher. The low weight allows clients with sessions on that server to return, but sends most new users to other servers. You may want to use this set up for a day and then disable the server with the previous (Server Enable checkbox) method. To see when it is safe to reboot a server, bring up the Site View view in Dispatch Manager and select Open Connections from the pull down menu in the middle of the screen. To display the number of open connections for the target server, place the cursor on the green box for that server. When the number is zero (0), you can reboot the server. Auto site restart If your site fails (during a power outage, for instance), the Central Dispatch auto site restart feature automatically restarts the site when the machines restart. (If the CDAction component is installed on the Central Dispatch node.) To disable the auto site restart feature, use the following procedures: Disabling auto site restart For a Solaris system: 1. Delete the file /etc/init.d/sentinel 2. Delete the file /etc/rc2.d/s99sentinel For an HP-UX system: 1. Delete the file /sbin/init.d/s676sentinel 2. Delete the file /sbin/rc2.d/s676sentinel For a Linux system: 1. Delete the file /etc/rc.d/init.d/sentinel 2. Delete the file /etc/rc.d/rc3.d/s99sentinel 2-42 Resonate Central Dispatch User Guide

85 Using Central Dispatch 3. Delete the file /etc/rc.d/rc4.d/s99sentinel 4. Delete the file /etc/rc.d/rc5.d/s99sentinel For an AIX system: 1. Delete the file /etc/rc.sentinel 2. Remove the sentinel /etc/inittab entry with the command: /usr/sbin/rmitab sentinel For a Windows NT or Windows 2000 system: Remove sentinel.exe from $RESONATE\bin. Exiting Dispatch Manager Exiting Dispatch Manager does not stop your Central Dispatch site. Exit Dispatch Manager 1. Click the Site Setup button. 2. Click the Operations tab. 3. Click the Disconnect From Site button. The Dispatch Manager welcome screen is displayed. 4. Close the window by selecting Close from the window manager menu (UNIX) or the window menu (Windows NT and Windows 2000). Useful environment variables You can use the following environment variables for tuning a site. Two of the variables tune agent performance by setting buffer sizes: RES_TCP_INBUF_SIZE sets the TCP Socket IN buffer size. Resonate recommends a size of 256K when using the Shadow Scheduling feature. Resonate Central Dispatch User Guide 2-43

86 Using Central Dispatch RES_TCP_OUTBUF_SIZE sets the TCP Socket OUT buffer size. Resonate recommends a size of 256K when using the Shadow Scheduling feature. The following variables set RXP timeouts for TCP services. The Central Dispatch RXP keeps TCP connections in tables and a timeoutdriven process cleans the tables at intervals. A connection is timed out and cleaned up from the table if the connection is long-lived and the connection receives no traffic for the length of the timeout value. Even if the connection is legitimate, the next traffic from the client after a connection is timed out will cause a reset. The default for all the timeouts (except for FTP_INACTIVE_CONN_ TIMEOUT) is 600 seconds. The default for FTP_INACTIVE_CONN _TIMEOUT is seconds (24 hours). SERVER_INACTIVE_CONN_TIMEOUT controls individual TCP connections for a server. Be sure that the value here is at least as long as the values for the scheduler variables below (except for FTP_INACTIVE_CONN_ TIMEOUT). With FTP_INACTIVE_CONN_ TIMEOUT, SERVER_INACTIVE_ CONN_TIMEOUT takes the scheduler timeout value. HTTP_INACTIVE_CONN_TIMEOUT controls individual TCP connections for HTTP and Cookie Persistence services for a scheduler. TELNET_INACTIVE_CONN_TIMEOUT controls individual TCP connections for Telnet service for a scheduler. PERSIST_INACTIVE_CONN_TIMEOUT controls individual TCP connections for IP Persistence service for a scheduler. FTP_INACTIVE_CONN_TIMEOUT controls individual TCP connections for FTP service for a scheduler. SSL_INACTIVE_CONN_TIMEOUT controls individual TCP connections for SSL services for a scheduler. The following variable enables port-independent persistent session rules (the shopping cart feature): RES_PORT_IND_PERSIST_SESSIONS enables the feature so that multiple requests from a client go to the same server, but can 2-44 Resonate Central Dispatch User Guide

87 Using Central Dispatch connect to different ports. Set this variable on every node that will host port-independent persistent sessions. This variable needs no assigned value. To set the variables on a UNIX system: 1. Using Dispatch Manager, disable the node. 2. Use the cdagentctl command to stop the Central Dispatch agents on a node. 3. Edit the file $RESONATE/bin/cdagentctl. 4. Add and export an environment variable. Follow the format of existing variables in the file. For example: RES_TCP_INBUF_SIZE=<value> export RES_TCP_INBUF_SIZE Where <value> is the size in bytes. 5. Use cdagentctl to restart the agents. 6. Using Dispatch Manager, enable the node. To set the variables on a Windows NT system: 1. Using Dispatch Manager, disable the node. 2. From Start, open Settings, Control Panel, System. 3. Open the Environment tab. 4. Add to the System Variables. For example: RES_TCP_INBUF_SIZE and your chosen value. 5. Reboot the system. 6. Using Dispatch Manager, enable the node. To set the variables on a Windows 2000 system: 1. Using Dispatch Manager, disable the node. 2. From Start, open Settings, Control Panel, System. 3. Open the Advanced tab. 4. Open Environment Variables. Resonate Central Dispatch User Guide 2-45

88 Using Central Dispatch 5. Add to the System Variables. For example: RES_TCP_INBUF_SIZE and your chosen value. 6. Reboot the system. 7. Using Dispatch Manager, enable the node. To check if the timeout variables are set, use the following command. If a particular timeout variable is not set, the command returns no value for that variable: rxpstat -A rxp_stats.txt Set up protection from attacks For a Central Dispatch site, you can configure protection from denial of service and bandwidth allocation attacks. Denial of service attacks Denial of Service attacks using the SYN-Bomb approach use the TCP connection initial synchronizing packet to tie up memory with connection table entries. The attack starts so many requests that all memory is taken up with uncompleted connections. Central Dispatch monitors the number of TCP connections waiting to complete. When the number of connections reaches a preset soft limit, Central Dispatch uses a much shorter timeout for dropping uncompleted connections. If the shorter timeout does not stop even more memory use, and the number of open connections reaches a preset hard limit, Central Dispatch chooses a pending connection to replace for each new connection. The soft limit changes the timeout from several minutes to less than one minute Resonate Central Dispatch User Guide

89 Using Central Dispatch Note When using IP Persistence and TELNET rules, Central Dispatch cannot shield a server from SYN-Bomb attacks. For servers using IP Persistence and TELNET rules, be sure to apply all SYN-Bomb protections recommended by the operating system vendor. Choosing a strategy The strategies for dealing with SYN-Bomb attacks are: Memory absorption: This method assumes that a system has enough memory to handle a SYN-Bomb attack plus handle legitimate traffic for the site. The memory usage soft limit and hard limit are effectively disabled by setting both the soft limit and hard limit to values that equal the system s physical memory. Hard limit only: Using the hard limit only ensures that as long as memory up to that limit is available, clients see no change in behavior. Even clients with slow connections (such as a normal time of 30 seconds between packets), would see no changes until memory usage hit the hard limit. For hard limit only, set the soft limit and the hard limit to the same value. Hard limit and soft limit: Using both limits creates some degradation in service, especially for clients with slow connections. The benefit is to possibly prevent memory usage from ever reaching the hard limit. The soft limit should be above the peak load of legitimate connections that you expect for the site. If the soft limit is too low, a system under normal load can tie up CPU usage by more frequent checking of connection timers. If possible, set the soft limit low enough that pruning connections during a SYN-Bomb attack happens early enough to never set off the hard limit. Resonate recommends setting the soft limit at 50% of the hard limit. Resonate Central Dispatch User Guide 2-47

90 Using Central Dispatch Setting Limits You can tune denial of service protection by setting the soft limit and hard limit for the number of connections allowed by a service. You can set limits for HTTP, FTP, IP Persistence, TELNET, SSL, and BTREE. Note The details given here about setting hard and soft limits ignore the many other factors that apply to tuning system memory. Memory used for connection entries is unpageable, so choosing limits must take into consideration the memory needed for other system tasks. The hard and soft limits for number of connections are set with environment variables. See On UNIX systems on page 2-50 and On Windows NT systems on page 2-51 for details on setting environment variables. The denial of service protection variables are: HTTP_CONN_HARD_LIMIT HTTP_CONN_SOFT_LIMIT FTP_CONN_HARD_LIMIT FTP_CONN_SOFT_LIMIT TELENT_CONN_HARD_LIMIT TELNET_CONN_SOFT_LIMIT PERSIST_CONN_HARD_LIMIT PERSIST_CONN_SOFT_LIMIT SSL_CONN_HARD_LIMIT SSL_CONN_SOFT_LIMIT BTREE_CONN_HARD_LIMIT BTREE_CONN_SOFT_LIMIT The default limits for number of connections are: OS/Service Soft Limit Hard Limit Hash Size Solaris/FTP 16,384 32,768 16, Resonate Central Dispatch User Guide

91 Using Central Dispatch OS/Service Soft Limit Hard Limit Hash Size Solaris/All other Services 65, ,072 65,536 Non-Solaris/FTP 4,096 8,192 1,024 Non-Solaris/All other Services 16,384 32,768 4,096 As each connection entry uses nearly 256 bytes, a Solaris system using the HTTP and FTP services would need the following memory to allow the default number of hard limit connections: 131,072 * 256 = 33 Mb for HTTP 32,768 * 256 = 8.4 Mb for FTP Total = 41.4 Mb The total is the maximum amount of memory that Central Dispatch RXP would request for connection tables. A system with 128 Mb of memory should run well with the default values, even during a denial of service attack. The hash size gives guidance for the maximum efficient hard limits. A hard limit of more than 8 times the hash size can degrade system performance. A hard limit of no more than 4 times the hash size is reasonable for machines with slow CPUs. You cannot reset the hash size. For Solaris systems, a change to the Sync Queue parameter can also help protect against SYN-Bomb attacks. Increasing the maximum size allows Solaris to keep accepting packets even while waiting to service current packets. Set the parameter to 128 if it is not already set higher. To set the parameter: 1. Edit the file /etc/system. 2. At the end of the file, add the line: sq_max_size= Reboot the system. Resonate Central Dispatch User Guide 2-49

92 Using Central Dispatch Bandwidth Allocation Denial Protection from bandwidth allocation denial (BAD) attacks happens as Central Disaptch monitors the amount of data received by a client through a single connection (by looking at the client ACKs received by the server). If that amount goes over your chosen setting, Central Dispatch lowers the packet size for all further data sent through that connection. Two environment variables enable this feature: BAD_PROTECT_FILE_SIZE is the maximum number of bytes allowed through a single connection before lowering the packet size. A value of 1000 means that after a total of 1000 bytes are acknowledged, the server sends all remaining data through that connection in smaller packet sizes. BAD_PROTECT_WIN_SIZE is the size in bytes for packets sent after a connection hits the limit in BAD_PROTECT_FILE_SIZE. A value of 100 means that the server sends data in packets of 100 bytes. If the BAD_PROTECT_FILE_SIZE is smaller than a client s TCP window size, a Central Dispatch server may send data well beyond the amount given for BAD_PROTECT_FILE_SIZE before lowering the packet size. The BAD_PROTECT_FILE_SIZE limit is set off only when the server receives a TCP ACK from the client for at least the amount of data equal to BAD_PROTECT_FILE_SIZE. On UNIX systems 1. Edit the file: /usr/local/resonate/bin/cdagentctl. 2. Add the variables BAD_PROTECT_FILE_SIZE and BAD_PROTECT_WIN_SIZE and set them to be exported. For example: BAD_PROTECT_FILE_SIZE=1000 export BAD_PROTECT_FILE_SIZE 2-50 Resonate Central Dispatch User Guide

93 Using Central Dispatch BAD_PROTECT_WIN_SIZE=100 export BAD_PROTECT_WIN_SIZE On Windows NT systems 1. From the Start menu, select Settings, Control Panel, System, Environment. 2. Add the System Variables BAD_PROTECT_FILE_SIZE and BAD_PROTECT_ WIN_SIZE and give them your chosen values. 3. Reboot the system. On Windows 2000 systems: 1. From the Start menu, select Settings, Control Panel, System, Advanced, Environment Variables. 2. Add the System Variables BAD_PROTECT_FILE_SIZE and BAD_PROTECT_ WIN_SIZE and give them your chosen values. 3. Reboot the system. Resonate Central Dispatch User Guide 2-51

94 Using Central Dispatch 2-52 Resonate Central Dispatch User Guide

95 CHAPTER 3 Monitoring Your Site Overview Once the Central Dispatch site is up and running, you can monitor site behavior. Dispatch Manager must be running for you to perform many of the tasks. This chapter describes the administrative functions available to someone with Central Dispatch management privileges. Someone with monitoring privileges can view activity and statistics but cannot manage configurations or shut down the site. Connecting to a running site To monitor or manage a Central Dispatch site, you must first connect to it or start it. When you start Dispatch Manager, the welcome screen prompts you to enter a node name (see Figure 3-1). Enter the name of any node in the site in the Connect to Node text box, then click OK. Dispatch Manager prompts you for a password (see Figure 3-2). Enter the monitor mode account password to just look at the site; enter the administration mode account password to both look at and make changes to the site. Resonate Central Dispatch User Guide 3-1

96 Monitoring Your Site Figure 3-1 Dispatch Manager welcome screen Figure 3-2 Password dialog 3-2 Resonate Central Dispatch User Guide

97 Monitoring Your Site Dispatch Manager then: Opens the Site View. Starts receiving data from the site. Begins revolving the Central Dispatch icon in the lower-right corner indicating a Dispatch Manager connection to a site. Connects to the primary Commander host, if one is configured. (To learn more about Commander, read the Resonate Commander User Guide.) During your Dispatch Manager session, you can disconnect from the site at any time. Click Site Setup, Operations, then click the Disconnect From Site button. The Welcome screen is displayed. You can now connect to another site if you want to. Note You can connect to a site whether it is running or stopped. Displaying site statistics When you first connect to a site, Dispatch Manager opens the Site View (see Figure 3-3). The names of the current schedulers at the site appear at the top of the window. (In the figure, the scheduler nodes are tungsten and rubidium.) The line graph in the upper half of the window displays collective statistics for the schedulers (including number of open connections and number of hits per second) If no scheduler is currently operating, a warning message appears in red in place of the name of the current scheduler. The bar graph in the lower half of the window displays real-time activity for either the nodes or the VIPs in the site. To change the view, Resonate Central Dispatch User Guide 3-3

98 Monitoring Your Site Figure 3-3 Site View select either Nodes or VIPs in the left-hand dropdown list above the bar graph. The Show All checkbox changes the size of the bars in the bar graph. When the box is not checked, if the nodes displayed take more space than is available in the view, a scroll bar appears below the node graph bars. When the box is checked, the node graph bars are compressed (which truncates the name in each bar) enough to show all nodes at once. The bar graph can display any of several performance statistics. To display different statistics, click the middle dropdown list and select one of the statistics listed in the table below. (Network Latency and CPU Load can be selected only if Nodes is selected in the left-hand dropdown list.) The bar graph rescales to maximize its dynamic range. 3-4 Resonate Central Dispatch User Guide

99 Monitoring Your Site Occasionally, a big spike may cause the graph to scale by a large amount. To reset the scale, use the dropdown list to reselect the statistic you want to view. Statistic Open Connections Hits Per Second Network Latency CPU Load Description Number of active network connections. Number of client requests received in the last second. (Not available for affiliated servers) Average elapsed time for a SYN packet to travel from a scheduler to a server plus the time needed for the SYN ACK packet to travel from the server to the scheduler. (In tens of microseconds.) Load on the computer. (Percent of utilization. This statistic mostly reflects load on the processor. However, this parameter also rises if the computer runs out of physical memory or swap space.) (Not available for affiliated servers.) Table 3-1 Site View statistics selection Note Since the bar graph averages statistics over two seconds, the bar graph statistics may vary slightly from the overall site statistics. Note The CPU Load statistic for a node goes up dramatically (possibly to 100%) when a screen saver is active on the node. Resonate Central Dispatch User Guide 3-5

100 Monitoring Your Site The bars in the bar graph display information either for nodes or VIPs. In the case of nodes, the color denotes the node s configuration and status, as described in the following table: Color Node Configuration Status Blue Green Dark Green Dedicated scheduler (not currently enabled as a server) Server or combined scheduler and server. Affiliated server. Responsive Responsive Red Scheduler, server, or both Not responsive Yellow Table 3-2 Disabled node (not currently enabled as either a scheduler or a server) Site View color code Responsive but disabled VIPs are always colored green. Note Displaying the bar graph value in the status bar When viewing the bar graph, you can position the mouse cursor over the node or VIP to see the numeric value of the statistic that the bar graph is displaying. This value appears in the status bar at the bottom of the window (below the row of buttons). Changing the bar graph order The right hand drop down menu above the bar graph determines the order in which nodes appear in the bar graph. The choices are: Schedulers First The default view that puts schedulers on the left. 3-6 Resonate Central Dispatch User Guide

101 Monitoring Your Site Sorted by Node Name Displays nodes by node name in alpha order starting from the left. Sorted by Alias Displays nodes in alpha order by node alias (if you use aliases) starting from the left. Single Group Shows only the nodes in a selected group. With this choice, a list of all the groups defined for the site appears at the far right of the panel. (Set up groups on the Groups tab of the Site Setup view.) Viewing node activity Display node activity 1. Click on Site View and select Nodes from the dropdown list. 2. Click on a node. The Node view displays information about a single node, with the node s hostname and IP address at the top, as shown below. The line graph displays the four node performance statistics, each in a different color. The statistics are Open Connections, Hits Per Second, Network Latency., and CPU Load. Note The statistics available for an affiliated server are Open Connections and Network Latency. 3. Click Node Information for information and status on the individual node. The status display contains the following information: Name: The node name. Node type: The node is a server, scheduler, or backup scheduler. Server state: The node is enabled as a server or disabled as a server. Connection state: The Central Dispatch agent at the node is online or offline. Agent connection state may indicate the state of Resonate Central Dispatch User Guide 3-7

102 Monitoring Your Site Figure 3-4 Site statistics the machine; if an agent suddenly appears offline, its machine is probably down. RXP Version: The version and build date of RXP that is running. Agent Version: The version and build date of the current agent. If this version is incompatible with versions other agents are running, you will receive an error message. If the node is a server, the scheduling rules that can direct requests to the node appear below the line graph. All rules that direct requests only to this node are highlighted in yellow to call your attention to a potential single point of failure requests that match only the highlighted rule will fail if this node fails, is taken off-line, or is disabled. 3-8 Resonate Central Dispatch User Guide

103 Monitoring Your Site If a node is both a scheduler and a server, the node type is scheduler and the server state is enabled. Viewing VIP activity Display VIP activity 1. Click on Site View and select VIPs from the dropdown list. 2. The VIP view displays information about all the VIPs (see Figure 3-5). Figure 3-5 Display of VIP activity The line graph displays the node performance statistics, each in a different color. These statistics are Open Connections and Hits Per Second. Resonate Central Dispatch User Guide 3-9

104 Monitoring Your Site 3. Click on any one of the VIPs. You now see the statistical information for just the view you selected in the upper half of the window (see Figure 3-6). In the lower half you see the scheduling rules for that VIP. Click the Next and Previous buttons to display the VIP view for the other VIPs you currently have defined at your site. Figure 3-6 Display of VIP activity for a single VIP Messages and agent logs Central Dispatch records all its activities in log files stored on each node in the Central Dispatch site. The pathname of the log file at each node is one of the following: installation directory/log/agent-dir.hostname/agent-log 3-10 Resonate Central Dispatch User Guide

105 Monitoring Your Site installation folder\log\agent-dir.hostname\agent-log Each agent running on a node writes messages to its own log file. This is a text file that you can view in an editor or browser program. Using Dispatch Manager you can save all of the agent logs to a single file (merged and ordered by timestamp) and you can clear all of the agent logs on all the nodes. Each agent also sends a copy of the messages it writes to its log file to Dispatch Manager. You can view these messages on the Messages tab of the Messages view in Dispatch Manager (see Figure 3-7 on page 3-12). Note Affiliated servers do not have a Central Dispatch agent and so do not generate any agent messages. In addition, Dispatch Manager generates its own messages and displays them on the Messages tab. These messages are not sent to the agents, so they do not appear in any of the agents log files. Because all the agents send their messages to Dispatch Manager, the list of messages in Dispatch Manager gives you a unified view of what is happening at your site. You can filter the messages to view only some of them (depending on which ones you are interested in), and you can generate and pager alarms for certain messages based on their severity. When a warning, error, or critical message is logged, Dispatch Manager displays an icon that shows the severity of the message to the right of the Messages button. This alerts you to a possible problem even if you are not currently viewing the Messages tab. You can save the list of messages displayed in Dispatch Manager to a file and you can clear the messages. You can also change the number of messages Dispatch Manager can display by changing the size of the Resonate Central Dispatch User Guide 3-11

106 Monitoring Your Site Viewing messages circular buffer Dispatch Manager uses to store the messages it receives from the agents. To view messages, click the Messages button, then the Messages tab. Warning Information Event Received Disabled Action Figure 3-7 Messages display On the Messages tab you see all the messages generated at the site since you connected to the site (unless you have cleared the messages) Resonate Central Dispatch User Guide

107 Monitoring Your Site For each message, the information listed in Table 3-3 appears. Column Severity Icon Description One of the following: Critical or error: A serious operational problem that might disrupt the operation of the site, or a problem that effects the overall performance of the site, such as a nonresponsive node or an agent s inability to configure a node. Warning: A generally non-harmful event that might indicate more serious problems. This includes errors in connecting to the backup host or detecting that the backup host is down. Information: Description of a normal event, such as addition of a new node or virtual IP address. Event received, but not yet performed. Action performed successfully. Disabled action. Failed action. Node Facility Table 3-3 Node on which the event occurred. Central Dispatch component, such as manager, agent, or RXP. Message list information Resonate Central Dispatch User Guide 3-13

108 Monitoring Your Site Column Subfacility Time Message Table 3-3 Description The part of the facility that generated the event. This information is useful for reporting problems to Resonate Technical Support. Day, date, and time when event occurred. Terse text description of the event. Message list information (continued) To see the details of a message, click that message s entry. An expanded description of the message appears in the Detail field. Filtering messages You may want to view only certain messages instead of looking at the whole log. For example, you could view only the messages originating from a particular node. To view only certain types of messages, click Messages, then click the Filters tab. Dispatch Manager opens the Filters dialog (see Figure 3-8). Set a filter 1. Click on one or more of the choices in any or all of the lists that specify Severity, Facility, Sub-facility, and Node filters. 2. Click Set Filter. The filtered list of messages appears below. To clear the messages currently displayed, click Clear Filter. Managing messages and agent logs Using Dispatch Manager you can save and clear all the agent logs. You can save and clear all the messages viewable on the Messages tab. Also, you can change the maximum number of messages you can view on the Messages tab Resonate Central Dispatch User Guide

109 Monitoring Your Site Figure 3-8 Filters dialog of the Messages view To do any of these tasks, click the Log Management tab in the Messages view of Dispatch Manager. You then see the following items on the tab: Clear Messages button Click this to immediately remove all messages from the Messages tab. Save Messages button Click this to save all the messages on the Messages tab to the file you specify. Clear History button Click this to empty all the agent logs and Commander logs in the site. The agents and Commander continue to write messages to their log files after you have cleared them. (For more information about Commander, read the Resonate Commander User Guide.) Resonate Central Dispatch User Guide 3-15

110 Monitoring Your Site Save History button Click this to save all the messages in all the agent logs in the site and the Commander logs to the file you specify. All entries are sorted by their timestamp. Max number of messages in buffer text box Enter the maximum number of messages that Dispatch Manager will display on the Messages tab. Once the maximum is reached, Dispatch Manager overwrites the oldest message whenever it receives a new one. Note You must hit the Return key after entering a new value for the maximum number of messages in the buffer. Agent log file format The format for agent log files is given in the following example: Agent 3 kernel RXP enabled The fields are separated by the pipe character. The first field is the time stamp in c-time. The second field is the facility sending the message. The third field is the message severity. A severity of 3 is an information message, 2 is a warning message, 1 is an error message, and 0 is a critical error message. The fourth field is the subfacility. The last field is the message text. Setting up alarms Central Dispatch can notify you when problems occur by sending an message or page. Dispatch Manager does not need to be running at the time Resonate Central Dispatch User Guide

111 Monitoring Your Site When the message you specify is logged, the Central Dispatch agent reporting the message executes an or pager alarm script. The agent logging the message is typically the agent on the node where the event about which the message is reporting occurred. However, if a node becomes unavailable, another agent reports the loss of the failed node. The reporting agent identifies alarms by severity, facility, and address. If you set up more than one alarm with identical severity, facility, and parameters, only one message is sent. Resonate supplies an alarm script for the mail program and a pager alarm script. The UNIX pager alarm script requires modifications read Using the pager alarm script on page This section describes how to: Edit the pager alarm script Create an alarm Test an alarm Modify or delete an alarm Using the pager alarm script In the RESONATE/bin directory you can find a sample shell script called pagealarm. This script is designed to work with pagers that use the Telocator Alphanumeric Protocol (TAP). Modify the script as appropriate to work with your own pagers. The text of the sample script is: MESSAGE="${2}:${3}" PAGEHOST="your-pager-host" # this is your paging host MODEMDEV="your-modem-device" # device for the modem ALPHA="a" NUMERIC="n" MODE=${ALPHA} # set to be alpha or numeric PIN="your-PIN" # if alphanumeric, enter your PIN here if [ "${MODE}" = "${ALPHA}" ] Resonate Central Dispatch User Guide 3-17

112 Monitoring Your Site then su uucp -c "rsh ${PAGEHOST} '/usr/local/bin/pager ${MODE} ${MODEMDEV} ${1} ${PIN} ${MESSAGE}'" else su uucp -c "rsh ${PAGEHOST} '/usr/local/bin/pager ${MODE} ${MODEMDEV} ${1} ${MESSAGE}'" fi exit 0 The script takes the following input arguments: Argument Description $1 The pager number of the recipient. You specify this message when setting up the alarm in Dispatch Manager. $2 The pager message to send. You specify this message when setting up the alarm in Dispatch Manager. $3 Alarm text. Table 3-4 Alarm script arguments Make the following edits: Keyword PAGEHOST MODEMDEV Description Replace the text your-pager-host with the name of a host that has a modem. If the local host has a modem, enter its name or use the keyword localhost. Replace the text your-modem-device with either /dev/cua/a or /dev/cua/b, depending on how the modem is defined to the system. Table 3-5 Alarm script edits 3-18 Resonate Central Dispatch User Guide

113 Monitoring Your Site Keyword MODE PIN Table 3-5 Description MODE specifies the pager type. The default is ALPHA (alphanumeric). Change the value to NUMERIC if necessary. If you are using an alphanumeric pager, replace the text your-pin with the personal ID number (PIN) that identifies you to the paging service. Alarm script edits (continued) Note After you edit the pager script on one system, copy it to all other systems where Central Dispatch is installed. The Central Dispatch installation script does not overwrite the alarm executable; this preserves your modifications if you reinstall Central Dispatch. Creating alarms If the alarm scripts that you want to use are in place, you can create an alarm. Central Dispatch has the ability to send mail alarms to a specified SMTP host. Create an alarm 1. Click Message, then click the Alarms tab to display the Alarms dialog (see Figure 3-9). 2. Select a Severity level. For a description of each severity level, see the table under Viewing messages on page Select a Facility. For a description of each facility, see the table under Viewing messages on page Enter the message you want to associate with the alarm in the Message field. Resonate Central Dispatch User Guide 3-19

114 Monitoring Your Site Figure 3-9 Alarms dialog of the Messages view Note Do not use quotation marks anywhere in the message. 5. Specify whether you want a page, an notification, or both: For a page alarm, click the By Page radio button and enter the pager number. Be sure to enter the phone number exactly as required. For example, you might need to prepend an outside line number such as 9, or to use the format 1-area_code-phone_number. For an alarm, click the By radio button and enter the address in the Address text box Resonate Central Dispatch User Guide

115 Monitoring Your Site Note If any of the nodes in your site are running Windows NT or Windows 2000, you must enter, in the Address text box, the complete hostname of the computer where the recipient receives mail. The name must include the SMTP mail server. For example, if sent to the webmaster account is actually received at iridium.giantcorp.com, where iridium is the mail server, list the address as You must do this even if can normally be addressed to that account as simply Testing alarms 6. Click Add to add the alarm. A text representation of the alarm appears below and the new alarm configuration is immediately propagated across the site. You can create any number of alarms. When testing, remember that the reporting agent identifies alarms by severity, facility, and address. If you set up more than one alarm with identical severity, facility, and parameters, only one message is sent. To test an alarm: 1. Click the Alarms tab in the Messages view of Dispatch Manager (see Figure 3-9). 2. Select the alarm in the Currently Specified Alarms box. 3. Click Test. Central Dispatch sends an or page message to the destination specified in the selected alarm. Resonate Central Dispatch User Guide 3-21

116 Monitoring Your Site Note You can test an alarm once every five minutes. Central Dispatch agents enforce a five minute interval between instances of the same alarm to ensure that a repeated error does not generate a flood of alarms. The test alarm message has the following information: Information Severity Node Facility Subfacility Message Value The alarm severity that you tested The node where you are running Dispatch Manager Manager Alarms Test Alarm Table 3-6 Alarm message components Modifying alarms To modify an alarm: 1. Click the Alarms tab in the Messages view of Dispatch Manager (see Figure 3-9). 2. Select the alarm in the Currently Specified Alarms box. The dialog updates to display the alarm information in the various fields. 3. Make the changes you want and click Update. The change takes effect immediately Resonate Central Dispatch User Guide

117 Monitoring Your Site Deleting alarms To delete an alarm: 1. Click the Alarms tab in the Messages view of Dispatch Manager (see Figure 3-9). 2. Select the alarm in the Currently Specified Alarms box. 3. Click Delete. The alarm is deleted immediately. Windows NT and Windows 2000 Performance Monitor The Windows NT and Windows 2000 Performance Monitor tool retrieves some data about the Central Dispatch RXP. In the Performance Monitor window, click on the plus sign (+) in the tool bar to Add to Chart. Table 3-7 shows the Object/Counter list for Central Dispatch. The first object column, RXPVNIC, is what you would see on a Central Dispatch server. The other three object columns show what you would see on a Central Dispatch scheduler. Object/ Counter RXPVNIC RXP Scheduler RXP Server RXP VIP CPU load Enabled Scheduler enabled Server priority Server weights Yes Yes Yes Yes Yes Yes Yes Yes Yes Total hits Hits /second Open connections Yes Yes Yes Yes Yes Yes Yes Yes Yes Table 3-7 Performance Monitor objects for a scheduler Resonate Central Dispatch User Guide 3-23

118 Monitoring Your Site Object/ Counter RXPVNIC RXP Scheduler RXP Server RXP VIP Retries Latency Errors Yes Yes Yes Table 3-7 Performance Monitor objects for a scheduler 3-24 Resonate Central Dispatch User Guide

119 CHAPTER 4 Using Scheduling Rules Resource-based Scheduling uses scheduling rules to assign incoming requests in a distributed Web site environment. These rules define which requests are sent to which nodes in your site, depending on the type of request (such as HTTP), the VIP to which the request is sent, and the kind of object requested (such as a Web page).this chapter describes the syntax of scheduling rules and what they mean. It also explains how to specify new rules using Dispatch Manager. Rule components When a client sends a request to any Internet site, it specifies a service (such as HTTP for retrieving a Web page), a hostname (resolved to a VIP by a DNS lookup), a port number (which, if not explicitly specified by the client, assumes some default value), and a path expression denoting the object the client is requesting (such as the pathname of a Web page). A scheduling rule may or may not have all of the components (see Table 4-1). A rule also specifies the servers that can respond to a client request, depending on the service, VIP, port number, COS group, and path expression or cookie parameter included in that request. Consider the rule whose components are shown in Table 4-1: Service VIP VPort SPort Time out COS Resource Server FailOver Server HTTP A intro.html gold-1 gold-2 Resonate Central Dispatch User Guide 4-1

120 Using Scheduling Rules Table 4-1 lists all of the components of a scheduling rule: Component Example Required/ Optional Internet service (protocol) HTTP Required VIP Required Virtual port number 80 Optional Server port number 81 Optional Timeout (SSL3, IP Persistence and Cookie Persistence) 0 Optional COS group A Optional Resource to look for (HTTP, SSL, and Cookie Persistence) List of eligible Central Dispatch servers Cookie or URL path expression (*/intro.html) gold-1 Required Required List of failover servers gold-1 Optional Table 4-1 Scheduling rule components Now assume the Central Dispatch scheduler receives the following request: The scheduler compares the request against the service, VIP, virtual port number, COS group, and path expression of any matching scheduling rule. Because there is a match, the scheduler transfers the connection from the client making the request to the gold-1 server on port 81. The COS entry determines the maximum number of open connections allowed on gold Resonate Central Dispatch User Guide

121 Using Scheduling Rules Note Even though no port number is specified in the example request above, the default HTTP port is 80. Therefore, the request does match the rule. Any combination of service name, VIP, virtual port number, and path expression or cookie attribute-value pair can be paired with a list of servers and server port numbers. For example, you can distinguish among requests based on service only, on service and port, or on service, port, VIP, and (in the case of HTTP or Cookie Persistence requests) path expression or attribute-value pair. COS Groups and Thresholds Each rule tab in the Scheduling Rules view includes a COS Group box with a pull-down menu. The choices in the menu are A, B, and C. The choices relate to a COS (class of service) level for the node referenced by each rule. Note that the classes are defined on the System Nodes tab of the Site Setup view. The Scheduling Rules view references the COS groups set up for the node in the Site Setup view. COS group definitions are tied to individual nodes in a site, not to VIPs. A rule can select a node only if the selection would not violate the COS limit for either the server or the port. Choices made with COS depend on the state of the server at the time of a client request. As traffic levels change the choices made by a rule with COS levels will change. By using only one COS group for a node or port, you can set connection thresholds. Each server node or port on a server node can have a maximum allowable number of connections specified. When the number of connections reaches that threshold, the scheduler directs no more traffic to that node or port. New requests can go to failover servers. Resonate Central Dispatch User Guide 4-3

122 Using Scheduling Rules See Class of service thresholds on page 1-28 for an explanation of COS groups. See Set up COS groups and thresholds on page 2-10 for setting up group values. Failover servers For all services, you can include a list of one or more failover servers with each rule. The scheduler targets failover servers when none of the regular servers in the Servers list is available. The regular servers may not be responding or may be at their COS limits for connections. Connections to failover servers are scheduled in the same way as with regular servers. That means that load balancing among available failover servers and COS limits for individual failover servers work the same way as with regular servers. Use the Failover Servers field to list failover servers. The syntax is similar to entries in the Servers field. The following example lists failover servers gold-1 and gold-2 and their valid ports. Note that each server entry is separated by a pound sign (#). The server IP address can be used in place of the server name. If you give no port, the default is the same port used in the Servers field. gold-1:80-85,87#gold-2:80-85,90-92 Kinds of scheduling Using scheduling rules you can configure the scheduler to make decisions based on the: Service requested VIP named in the request Port to connect to Pathname of requested object (HTTP requests only) Attribute-value pair (Cookie/CGI requests only) Need for persistent sessions 4-4 Resonate Central Dispatch User Guide

123 Using Scheduling Rules Service-based scheduling Central Dispatch can schedule requests to the HTTP, FTP, Telnet, IP Persistence, SSL3, and Cookie/CGI Persistence services. Note that the HTTP service can schedule by URL, Cookie, or CGI parameters. Service-based rules are useful for Web sites in which some servers contain HTTP-related documents, others serve an FTP site, still others allow Telnet logins, and so on. The following are examples of service-based rules: All FTP requests are served by actinium. All HTTP requests are served by vanadium. Note Central Dispatch keeps TCP-based services requests in connection tables. The Central Dispatch RXP times out inactive entries in the tables through default timer values or through environment variables that you can set. If your site has long-lived TCP connections with long periods of no activity, you may want to increase the timer values. See Useful environment variables on page 2-43 for details. VIP-based scheduling To schedule requests for a VIP to various servers, you can specify the VIP in a scheduling rule. The following are examples of VIP-based rules: All requests to are served by actinium. All requests to are served by vanadium. Resonate Central Dispatch User Guide 4-5

124 Using Scheduling Rules Port-based scheduling If your site uses multiple ports for a service, you can create separate rules for the different port numbers. The following are examples of port-based rules: All HTTP requests on port 80 are served by vanadium. All HTTP requests on port 81 are served by actinium. Port mapping You can also map the port in the client s request (the virtual port) to a different port on the server (the server port). For example, you can define a rule such that HTTP requests on port 80 are serviced by vanadium on port This port mapping is totally transparent to the client. As far as the client can tell, it has made a connection to port 80 (on the VIP specified in its request). One restriction on port mapping is that requests that are mapped to a different port require that all servers eligible to service the requests listen on the same server port. This restriction applies to all requests that specify the same VIP, the same virtual port number, and the same path expression. For example, assume you take all HTTP requests to the VIP on port 80 for HTML files named intro.html and map them to port 81. All the eligible servers you list in the scheduling rule must listen on the same port (in this example, 81). What you cannot do is take HTTP requests to on port 80 for HTML files named intro.html and map them to port 81 if the scheduler selects the server actinium, and to port 82 if the scheduler selects the server vanadium. You must specify all of the eligible servers in the same rule, and they must all listen on the same port (in this case, port 81). 4-6 Resonate Central Dispatch User Guide

125 Using Scheduling Rules However, you can create port mapping rules that are serviced by different servers listening on different server ports provided the rules differ in at least one of the following fields: VIP Virtual port number Path expression Attribute-value pair Port load balancing Port load balancing directs client requests for a port to any one of a set of identical processes listening on different ports on a particular server. During port load balancing, Central Dispatch has already determined which server should get the request. Central Dispatch applies a roundrobin algorithm to determine which port to choose on the server selected by applying the scheduling rules. If the connection to the selected port fails, the scheduler applies the rules again to select a server for that request, rather than trying the next available port on the original server. In port load balancing, the most-recently-used port is tracked by rule and not by port. This means that if there is more than one rule which specifies the same port, that port may receive multiple requests at the same time because those requests were directed to the same port by applying different scheduling rules to those requests. Combining port mapping and port load balancing To enter multiple virtual ports and server ports on the scheduling rules tabs (HTTP, FTP, Generic TCP, SSL3, IP Persistence, Cookie Persistence) to take advantage of port mapping and port load balancing, enter port numbers separated by commas and port ranges separated by a hyphen. For example the following entry in the Server Port field selects all the ports between 80 and 90 except for ports 82, 84, 85, and 86: 80,81,83,87-90 Resonate Central Dispatch User Guide 4-7

126 Using Scheduling Rules Path-based scheduling Path-based rules allow you to direct an incoming request to a specific server based on the path expression in the URL specified in a client s HTTP request. You can direct requests to particular servers based on the filename extension, the directory where the file is located, or even the complete name of the file. Note Path-based scheduling applies only to HTTP service requests. The following are examples of path-based rules: HTTP requests to test160 for a document whose path expression includes */cgi/* are served by actinium. HTTP requests to test161 for a document whose path expression includes *.jpeg (that is, requests for JPEG files) are served by thulium and vanadium. HTTP requests to host test164 for a document whose path expression includes */myfile.html are served by vanadium. HTTP requests to host test163 for a document whose path expression includes */myfile.html are served by vanadium. HTTP requests to any host for a document whose path expression includes ~* are served by actinium. Root directory URL pathnames descend from a Web server directory called the HTTP root. The location for the HTTP root directory is specified as the document root in the HTTP server configuration file. To ensure matches between scheduling rules and incoming requests, path expressions in scheduling rules must also be relative to the HTTP root. Consider an example Web resource with the pathname: 4-8 Resonate Central Dispatch User Guide

127 Using Scheduling Rules /mydir/subdir/subsubdir/myfile.html The leading slash means that the pathname starts from the HTTP root directory. Note A set of scheduling rules that use path expressions must have at least one path of asterisk (*) or at least one path starting with a root slash (/). Without one or the other, the VIP is erroneously seen as down. Path expression syntax You specify pathname-based scheduling by using a restricted regular expression called a path expression. Path expressions are designed to match the syntax of URL pathnames. A path expression can consist entirely of literals (directory names and file names) or a combination of literals and wildcards (but only in certain legal combinations). A literal path expression matches only a client request for a directory of file that exactly matches that literal. For example, the path expression /mydir in a scheduling rule matches all and only those requests for the directory mydir under the HTTP root directory. Similarly, the literal path expression / matches all and only those requests for the root directory. Note Because Central Dispatch uses a fast pattern-matching algorithm to achieve high performance, it does not support arbitrary UNIX regular expression syntax. For example, you cannot use the question mark (?) as a wildcard symbol in path expressions. Resonate Central Dispatch User Guide 4-9

128 Using Scheduling Rules Wildcards The path expression component of a scheduling rule consists of the following tokens: Directory names File name (less the extension) File name extension You can use the asterisk (*) wildcard character in a path expression to match some (but not all) of the tokens and token combinations in a URL pathname. Table 4-2 lists some of the valid and invalid uses of the * (asterisk) wildcard in path expressions. It contains examples of path expressions containing wildcards. The table lists the meaning of each wildcard and describes the files that would match each path expression. The most general form of a path expression with wildcards is: */directory_names/*/*.file_extension In this form you can match all files with the specified file extension located in all directories whose path name includes directory_names as a substring anywhere in the path name. Note As Table 4-2 shows, you cannot use the * (asterisk) wildcard to denote an arbitrary file extension. Cookie/CGI-based scheduling The HTTP and Cookie Persistence services allow you to direct an incoming HTTP request based on an Attribute-Value Pair. The attribute-value pair field appears with the HTTP tab when you choose a Resource of Cookie, CGI Parameter, or Cookie or CGI 4-10 Resonate Central Dispatch User Guide

129 Using Scheduling Rules Wildcard Usage Example Path Expressions Valid All file names in the web root directory less the extension. All file names in the web root directory and subdirectories less the extension. One or more directories (either leading or embedded) File name less the extension Complete filename, including the extension A combination of one or more directories and a filename All the files in a directory. Also, any subdirectories of or files in the subdirectories of the specified directory Filenames and directories that begin with the specified literal string Filenames and directories that include the specified literal string Filenames that have a specified extension and whose pathname (except for the name of the file preceding the extension) includes the specified literal string *.html */*.html /mydir/subdir/*/myfile.html or */subsubdir/myfile.html /mydir/subdir/subsubdir/*.html /mydir/subdir/subsubdir/* /mydir/subdir/* /mydir/* (This matches the pathnames /mydir/file1.html, /mydir/subdir and /mydir/subdir/file2.html.) /mydir/feature* (This matches both the file /mydir/feature.html and the directories /mydir/feature_dir and my_dir/feature_dir/subdir.) */somedir/mydir/feature* (This matches any pathname that includes the substring /somedir/mydir/feature anywhere in the path.) */somedir/mydir/feature*/*.gif (This matches all files with the gif extension where the pathname of any directory containing such files includes the substring /somedir/mydir/feature anywhere in the pathname.) Table 4-2 Valid and invalid uses of wildcards in rules Resonate Central Dispatch User Guide 4-11

130 Using Scheduling Rules Wildcard Usage Example Path Expressions Tilde directory (beginning with a tilde) ~* (When using this notation, you cannot specify anything after ~* in the path expression.) Invalid Part of the file name that precedes the extension A directory embedded between two other directories More than one asterisk following the literal portion of the path expression, and the last asterisk does not replace the file name (less the extension) More than two asterisks following the literal portion of the path expression, and last asterisk replaces the file name (less the extension) File extension my*.html /mydir/*/subsubdir/myfile.html /mydir/subdir*/*/ /mydir/subdir*/*/*.gif myfile.* Table 4-2 Valid and invalid uses of wildcards in rules (continued) Parameter. The same choices appear on the Cookie Persistence tab in the Resource pull-down menu. With a rule set up from the HTTP tab, Cookie or CGI requests are directed to whatever server is available for the given rule. A sequence of connections from the same client do not necessarily go to the same server. With a rule set up from the Cookie Persistence tab, the same sequence would go to a single server. All rules using a Resource of cookies or CGI include an Attribute- Value Pair field. The attribute-value pair are separated by the equal sign (=), such as user-id=jsmith. The value portion (=jsmith in the example above) is optional Resonate Central Dispatch User Guide

131 Using Scheduling Rules The value portion can be an asterisk (*). A stand-alone asterisk represents all values. An asterisk with other characters is taken as a literal. For example, user-id=* in a rule matches client cookies of user-id=jsmith, user-id=jjones, and so on. The value portion can not have spaces. To handle a space from a browser or CGI script, use the plus character (+) in place of the space. For example, user-id=jsmith+company would handle a browser form entry of jsmith company. The attribute portion can be an asterisk (*), but the asterisk is treated as a literal character, not a meta-character. A rule with only an attribute portion will match a cookie only if the cookie contains an attribute and no value portion. For example, user-id in a rule will not match a cookie of user-id=jsmith. The attribute is case-insensitive, but value is case-sensitive. The attribute and value portions can not have spaces. For the value portion to handle a space from a browser or CGI script, use the plus character (+) in place of the space. For example, user-id=jsmith+company would handle a client cookie of user-id=jsmith company. The attribute and value can be any characters except a control character or special character. Special characters are: ( ) < ; : \ / [ ]? = { } Other characters, such as an ampersand (&), must be represented with a percent sign (%) and ascii value. For example, to handle jsmith&company, the value portion of the attribute-value pair would be jsmith%26company. To find ascii values on a UNIX system, use the command: man ascii. From a Windows system, do a web search on ascii chart. A rule using Cookie looks for the cookie argument in a cookie response header and matches that to the Attribute-Value Pair. A rule using CGI Parameter matches the CGI parameter list, which is passed with a URL immediately after the?. For example, an Resonate Central Dispatch User Guide 4-13

132 Using Scheduling Rules Attribute-Value Pair of ssno=* would match the following request line: /cgi-bin/printenv?ssno= A rule using Cookie or CGI Parameter causes the rule to look for a cookie argument or a CGI parameter in a request. The cookie argument takes precedence over the CGI parameter, meaning that if the rule finds a cookie argument, the rule looks no further. The rules for duplicate attribute-value pairs are: A Cookie rule and a CGI Parameter rule can have the same Attribute-Value pair. A client request with CGI parameters will be directed to the CGI Parameter rule. A client cookie request would get passed to the Cookie rule. A Cookie or CGI Parameter rule cannot use an Attribute Value pair that is used in either a Cookie rule or a CGI Parameter rule. Persistent service scheduling When you create persistent service scheduling rules, clients who make a sequence of connections to your site always connect to the same server rather than to different servers with each connection. You configure this sort of persistent session behavior by creating scheduling rules on the IP Persistence or Cookie Persistence tabs of the Scheduling Rules view. For each rule, you can set a Timeout parameter. The parameter directs any new connections from a client to the same server until the timeout period expires. Persistent service scheduling rules are useful at sites with CGI (or similar) applications that rely on locally stored information for processing client requests. You should create persistent service scheduling rules whenever servers maintain state information over multiple connections or sessions. By scheduling successive connection requests to the same server, Central Dispatch ensures that 4-14 Resonate Central Dispatch User Guide

133 Using Scheduling Rules the server maintaining the client s state data is always the server that is chosen to handle the client s connection request. Central Dispatch offers the following persistent services: IP Persistence is based on the client IP address, VIP, and virtual port. The Timeout clock starts when a client s last TCP connection closes. This service is for HTTP and SSL2 requests. Cookie Persistence stores the cookie or CGI attribute-value pair and VIP from a client s HTTP requests. The Timeout clock starts when a client s last TCP connection closes. SSL3 stores the 32 byte SSL session ID and VIP. The Timeout clock starts when the connection is created. Shadow Scheduling See Shadow Scheduling on page 2-20 for this persistence-related feature. Guidelines Note the following guidelines when using persistence services: SSL 2.0 servers only If you are running only SSL 2.0 servers (no SSL 3.0 servers), you should use the IP Persistence service to direct client connections to those servers. If traffic is only or mostly SSL2, you gain performance by using IP Persistence. Mixture of SSL 2.0 and SSL 3.0 servers If you have a mixture of both SSL 2.0 and SSL 3.0 servers and you must specify one or more of each type of server in the same scheduling rule, you can use the SSL3 service, not the IP Persistence service. However, session IDbased scheduling is disabled. Although SSL3 scheduling supports both SSL 2.0 and SSL 3.0 servers, supporting both types of servers exacts additional overhead. You will have better performance if you avoid mixing both protocols, that is, if all of your SSL servers for the rule use the same protocol (SSL 2.0 or SSL 3.0). Resonate Central Dispatch User Guide 4-15

134 Using Scheduling Rules To avoid the performance impact of mixed SSL protocols, you can create separate scheduling rules by defining distinct VIPs or ports for the rules that target your SSL servers. In that case, specify only SSL 2.0 servers in the IP Persistent service and only SSL 3.0 servers in the SSL3 service. Port-independent persistent sessions If you need to support port-independent persistent sessions, you must use the IP Persistence or Cookie Persistence services (not the SSL3 service). This is true even if some or all of the servers specified in the rules are running SSL 3.0. For details, read About port-independent persistent sessions on page To enable port-independent persistent sessions, see Useful environment variables on page Note SSL scheduling is the default configuration and applies to SSL 3.0 servers. If all servers at your site are running SSL 3.0 and you do not require port-independent persistent sessions, create SSL3 service scheduling rules for those servers. (This is because SSL3 service rules base scheduling decisions on session IDs.) For more information, read SSL3 scheduling on page Persistent service scheduling and Global Dispatch If you use the IP Persistence or Cookie Persistence services and are running Resonate Global Dispatch to distribute client requests across geographically separate Central Dispatch sites, you must configure Global Dispatch to run its own version of persistent sessions. Read the Global Dispatch User s Guide for more information about how to configure persistent sessions in that product. Multiple Proxy Servers and HTTP Persistence A site that receives client requests from multiple proxy servers can use the Subnet Mask-Based Persistence feature to allow HTTP persistence with the multiple proxy servers. The feature directs all IP addresses coming from a single subnet to the same server Resonate Central Dispatch User Guide

135 Using Scheduling Rules Subnet Mask-Based Persistence applies a subnet mask to the source IP addresses in the persistence table. For example, a subnet mask of sets persistence for all clients with IP addresses up to that value. To set the mask, open the Policies tab of Dispatch Manager s Site Setup view and enter the mask into the Persistent Subnet Mask field. Setting the persistent timeout parameter Some HTTP web servers try to reduce the overhead of starting a session with a client by keeping a cache of recent session information. If a client closes all of its connections to a server, then returns later to that same server to establish a new session, the server can by-pass the overhead of establishing the new session by fetching the saved information and re-using it. To take advantage of this feature, set a non-zero timeout in your persistent scheduling rules. If the client reconnects to your site before the timeout expires and uses the same identifiers it used previously, the scheduler reconnects the client to the same server and resets the timeout. See Persistent service scheduling on page 4-14 for the identifiers used by each type of persistent service. Scheduler session entries are not shared between different primary schedulers. If you are using Global Dispatch to send requests to multiple primary schedulers, the IP Persistence and Cookie Persistence services require that the client request be sent to the same scheduler. To ensure this occurs, set the persistent session timeout value in Global Dispatch to a value greater than or equal to the value used in Central Dispatch. Read the Global Dispatch User s Guide for more information. Specify the timeout (in seconds) on the SSL3, IP Persistence and Cookie Persistence tabs of the Scheduling Rules view. The default timeout is 120 seconds. Resonate Central Dispatch User Guide 4-17

136 Using Scheduling Rules About port-independent persistent sessions Port-independent persistent sessions control whether the client connections continue to go to the same server node even if the client specifies different port numbers in the connection requests. This feature is typically used to implement e-commerce applications. Using port-independent persistent sessions changes the logical level at which the scheduler keeps the persistent session table. For persistent sessions without port-independence, the scheduler keeps a persistent sessions table for each scheduling rule. Each connection through different rules can then be load balanced independently (by port) and possibly go to different servers in a VIP. With port-independence, the scheduler keeps a persistent sessions table for each VIP, meaning that all the scheduling rules on that VIP share the same table. A client making connections through more than one port (more than one rule) will get the same content server for each connection. Note For port-independent sessions to remain persistent to the same server, each rule for the VIP must include the exact same list of servers. If one rule does not contain the same list, sessions can get reset to different servers. To enable port-independent persistent sessions, see Useful environment variables on page You can configure port-independent persistent sessions for both SSL 2.0 and SSL 3.0 servers. When configuring port-independent persistent sessions for SSL 3.0 servers, Central Dispatch ignores the port specified in the IP Persistence and Cookie Persistence services. Port-independent persistent sessions are useful for electronic commerce applications (such as an electronic shopping cart). When a client shops electronically, for example, it might connect to one port to find and select the items to be bought, then connect to another port to communicate with a secure server to provide credit card information 4-18 Resonate Central Dispatch User Guide

137 Using Scheduling Rules SSL3 scheduling for the final purchase. This is useful because it limits the higher overhead of a secure connection to the very last portion of the client interactions with the e-commerce site (namely, the checkout and purchase phase). Despite having specified two different ports on the same VIP in its connection requests, the client continues to connect to the same server node in the Central Dispatch site. This server node can then maintain client purchase information throughout the entire session, while running different server programs (both insecure and secure, listening on different ports) that process different phases of the client s transactions. These phases include such transactions as selecting items to purchase, placing them in an electronic shopping cart, and finally purchasing the selected items. By scheduling successive connection requests to the same physical machine (even though the requests specify different port numbers to access the different server programs required to process the entire transaction), Central Dispatch ensures that all the server programs running on the same physical machine can access the accumulated state data for a particular client. SSL3 service scheduling allows Central Dispatch schedulers to make intelligent scheduling decisions by leveraging certain features of the SSL 3.0 protocol. In SSL 3.0, each session is identified by a unique, non-encrypted session ID that the schedulers use to make intelligent scheduling decisions. SSL3 provides a persistence protocol that lets servers maintain state information over multiple connections or sessions. By basing persistence on session IDs rather than client IP addresses (as is the case with IP Persistent service scheduling rules), the schedulers can unambiguously and reliably identify distinct sessions, even when they originate from the same client IP address. Resonate Central Dispatch User Guide 4-19

138 Using Scheduling Rules Since a Central Dispatch scheduler is able to distinguish unique sessions using the session IDs, it will by default schedule successive connection requests to the same server. The default SSL port is 443. The scheduler directs all client connection requests to the same server that is maintaining the client s state data. You configure SSL3 service scheduling rules on the SSL3 tab of the Scheduling Rules view Dispatch Manager. Guidelines Note the following guidelines when defining SSL3 service scheduling rules: SSL3 service scheduling rules are intended to be used with servers running SSL 3.0. These servers keep entire sessions secure from start to finish. Performance suffers when you define SSL3 service scheduling rules for servers running SSL 2.0. For this reason, if all of your SSL servers are running version 2.0 of the protocol, you should define IP Persistent service scheduling rules for those servers, not SSL3 service scheduling rules. For more information, read Persistent service scheduling on page Use SSL3 service scheduling rules if some of your servers are running SSL 2.0 and some are running SSL 3.0, and you cannot or choose not to create separate rule sets for these two different types of servers. Note Although the SSL3 scheduling service supports SSL 2.0 servers, the scheduling service incurs processing overhead to support it. Because this degrades system performance, you are urged to keep your servers homogeneous using either the SSL 2.0 or the SSL 3.0 protocol. If you are running both SSL 2.0 and SSL 3.0 servers and have not created separate rule sets for each type of server, a client running 4-20 Resonate Central Dispatch User Guide

139 Using Scheduling Rules SSL 3.0 may end up being connected to one of the servers running SSL 2.0. SSL3 service scheduling rules do not support port-independent persistent sessions. Such sessions maintain connection requests from the same client machine or IP address to the same VIP, even if the port specified changes from one request to another. If you require this type of support, no matter what protocol you are using for secure Web server software, you must configure your server to use IP Persistent service scheduling. For more information, read About port-independent persistent sessions on page If you create SSL3 service scheduling rules and are running Resonate Global Dispatch to distribute client requests across geographically separate Central Dispatch sites, you must configure Global Dispatch to run its own version of persistent sessions. Read the Global Dispatch User s Guide for more information about how to configure persistent sessions in that product. Setting the timeout parameter To reduce the overhead of starting a session with a client, you may have set up your server to cache recent session information. If a client closes all of its connections to a server, then returns later to that same server to establish a new session, the server can by-pass the overhead of establishing the new session by fetching the saved session information and re-using it. To be sure you are taking full advantage of this feature, you need to set the SSL3 service scheduling rules timeout parameter to a setting that is well-synchronized with the timeout setting for the server software that will respond to the client requests. For example, Web servers responding to a client request send session IDs to the client (in this case, a Web browser). When the client with the same session ID issues another request, its connection request is directed to the same server that serviced the previous request. This continuity of connections with the same server is known as an SSL Resonate Central Dispatch User Guide 4-21

140 Using Scheduling Rules session. However, Web servers have timeouts that they use to expire session IDs that are stale (too old). In SSL 3.0 the first data packet from the client does not contain the session ID. The server generates the session ID in response to a client message and sends the session ID to the client. The Central Dispatch scheduler stores this session ID, then starts its own session ID timeout countdown. The scheduler uses the session ID to keep track of the session between the client and the particular server the client is repeatedly connecting to. The scheduler also keeps track of how much time is left in its timeout before it expires the session ID. The timeout maintained by the Web server and the timeout maintained by the Central Dispatch scheduler serve two different purposes. However, these two timeout values determine whether or not a client request is maintained within a single SSL session. To preserve the integrity of the sessions, you must coordinate both timeout values to get the results that you expect. For example, if the SSL3 service scheduling rule timeout value is less than the timeout on the Web server, the session ID tracked by the Central Dispatch scheduler expires first. This means that the existing SSL session is no longer available. As a result, the scheduler can direct the next client connection request to a different Web server. On the other hand, if the SSL3 service scheduling rule timeout value is greater than the timeout on the Web, the session ID on the Web server expires before the session ID tracked by the Central Dispatch scheduler. When the client sends its next connection request, which includes the old session ID, the Central Dispatch scheduler directs the request to the same Web server as before. However, the Web server no longer honors the old session ID, since it has already expired. To avoid abnormal persistence behavior, set the SSL3 service scheduling rules timeout value slightly greater than the Web server (or other server software) timeout. You specify this timeout (in seconds) 4-22 Resonate Central Dispatch User Guide

141 Using Scheduling Rules on the SSL3 tab of the Scheduling Rules view of Dispatch Manager. The default timeout is 120 seconds. How Central Dispatch chooses a rule When the scheduler receives a connection request, it examines the URL and selects all matching scheduling rules. In some cases more than one rule might match. When more than one rule matches, the scheduler chooses the most specific rule. The most specific rule could be, but is not necessarily, the rule with fewest wildcards. Table 4-3 illustrate rules that vary from the more general to the more specific: Path Expression /my_dir/subdir/subsubdir/* /my_dir/subdir/subsubdir/ /my_dir/subdir/subsubdir/page.html Matching Resource Any file in the directory subsubdir or in any directory under the subsubdir tree. Any file in the directory subsubdir. The file page.html in the directory subsubdir. Table 4-3 Example path expressions and their meanings You can use this rule precedence feature to first create general rules, then override them as needed with specially-tailored exceptions. For example, assume you want three nodes (actinium, vanadium, and thulium) to respond to all CGI requests. Assume also a single exception: all requests that include secret.cgi in the pathname are to be serviced by thulium because all secret.cgi programs access private files that require controlled access (they are all stored just on thulium). Table 4-4 shows the path expression component and the Resonate Central Dispatch User Guide 4-23

142 Using Scheduling Rules eligible servers component of two different scheduling rules to accomplish this. Rule Path Expression Eligible Servers 1 */*.cgi actinium, vanadium, thulium 2 */secret.cgi thulium Table 4-4 Specific scheduling rule overrides more general one When a URL that includes secret.cgi in the pathname arrives, the scheduler matches the URL to both Rule 1 and Rule 2. The scheduler chooses Rule 2, the more specific rule, and transfers the connection to server thulium Specifying scheduling rules Specify scheduling rules 1. Click the Scheduling Rules button at the bottom of the Dispatch Manager window. The Scheduling Rules view appears. Tabs are activated for services that you have licensed. If you have licensed all services, you can select any of the tabs: HTTP, FTP, Generic TCP, SSL3, IP Persistence, or Cookie Persistence. Otherwise, you can select only the tabs for the services you have licensed. Note If the Scheduling Rules button is disabled, click the Site Setup button, then click the Policies tab and check the current scheduling policy. You must select the Resource-based policy before you can create scheduling rules Resonate Central Dispatch User Guide

143 Using Scheduling Rules Figure 4-1 HTTP dialog in Scheduling Rules view 2. Click on the service for which you want to create a rule: HTTP, FTP, Generic TCP, SSL3, IP Persistence, or Cookie Persistence. The service dialog displays the default rule. The default resourcebased rule directs all requests for all services to all servers. 3. In the VIP list box, select the one VIP to which this rule applies or select the asterisk (*) to apply the rule to all VIPs. 4. In the Servers list box select the server or servers to which this rule applies or select the asterisk (*) to apply the rule to all servers. 5. If you use a non-standard port number or range of ports for the service, put that number or range in the Virtual Port field. 6. To connect clients to a port or range of ports other than the one specified in the connection request (port mapping), specify this Resonate Central Dispatch User Guide 4-25

144 Using Scheduling Rules port in the Server Ports field. (For more information, read Port mapping on page 4-6.) 7. Choose a COS Group if you are using class of service levels. See Class of service thresholds on page 1-28 for COS information. 8. List Failover Servers if you want failover protection. 9. If you are creating an HTTP or Cookie Persistence rule, choose a Resource. 10. If you are creating an HTTP or Cookie Persistence rule, enter a path expression in the Path Expression field or a Cookie attribute/value combination in the Attribute-Value Pair field: If the target resources and content are fully replicated across all servers in the site, put the asterisk (*) wildcard character in the expression field. If the target resources and content are not fully replicated across all servers in the site, enter the expression that describes the resources or content to which this rule applies. 11. If you are creating an SSL3, IP Persistence, or Cookie Persistence rule, enter the length of the timeout (in seconds) in the Timeout field. 12. Click Add. Dispatch Manager displays the scheduling rules once you have created them (see Figure 4-2). Dispatch Manager displays scheduling rules in a tabular format with these columns: Service VIP V Port S Port Timeout COS Group Cookie Servers FailOverServers The column V Port stands for Virtual Port or port range; the column S Port stands for Server Port or port range. For example, assume you define a rule where server actinium handles HTTP requests directed to test164 on port 80. Assume further that this rule applies only to requests for file index.html in the document 4-26 Resonate Central Dispatch User Guide

145 Using Scheduling Rules Figure 4-2 Scheduling rules display root directory and that the connection actually goes to port 81 on actinium. After you have defined such a rule, Dispatch Manager displays a line in the Scheduling Rules list with information similar to the following: HTTP test A /index.html actinium Note For services (such as HTTP) where there is no Timeout value defined, the Timeout (fifth) column shows zero. Resonate Central Dispatch User Guide 4-27