StarTeam Performance and Scalability Techniques

Transcription

1 Performance and Scalability Techniques Guidelines for Planning, Configuring, and Tuning Repositories A Borland White Paper By the Development Services Platform Team December, 2003

2 Contents Overview Architecture Overview Hardware and Capacity Planning Server Component Distribution Options Shared Server Machines Large Configurations Distributed Over Multiple Servers Multiple Configurations Sharing a Database Server Server Component Resource Factors Server Process Hardware Factors Database Hardware Factors Vault Hardware Factors MPX Message Broker Hardware Factors Usage Performance Factors Database Capacity Planning Vault Capacity Planning Server Component Hardware Recommendations and MSDE on the Same Machine Server Alone Database Server Hardware Recommendations MPX Server Use MPX for Distributed Teams Use Multicast for Large LANs Use MPX Profiles for Multiple Messaging Services Performance Tuning Server Tuning Techniques StarbaseServer.ini File Options Server Configuration Options Vault Tuning Techniques Database Tuning Techniques General Database Tuning Techniques SQL Server Database Tuning Factors Glossary Appendix - RAID Systems RAID 0: Striping with No Parity RAID 1: Shadowing and Mirroring RAID 0+1: Mirroring on two sets of Striped disks RAID 5: Striping with Rotating Parity

3 1 Overview is a software configuration management (SCM) system that serves a wide range of application development environments. If your team is small, you can install a server on a single machine, use the default database (MSDE), and begin using it effectively with little administration. However, if your organization has hundreds of users and expects high concurrency, you have many choices for addressing scalability and performance. This article provides you with guidelines for planning, configuring, and tuning your deployment. This information is based on successful practices of s largest installations. This article is intended for administrators, project managers, and developers who plan, deploy, and maintain server configurations. This information is directed at performance and scalability issues. See the separate article Configuration Best Practices for guidelines on how to configure projects and views within a server to support your organization s development practices. The information in this article is organized into the following sections: Architecture Overview: This section provides an overview of the client/server architecture and describes basic factors related to performance and capacity planning. Hardware and Capacity Planning: This section helps you plan the hardware configuration needed to support your organization s capacity and concurrency requirements. Hardware options for distributing the server, database, and vault components are discussed. Capacity planning suggestions specific to Oracle and SQL Server databases are provided. MPX: This provides an overview of MPX, which can be added to a deployment to increase server scalability and enhance client throughput. Options for distributing MPX Message Brokers and addressing mixed internal and external access are discussed. Performance Tuning: This section describes tuning parameters and techniques with which you can optimize the performance of your implementation. The information in this article is relevant to the 5.3 release. In addition to the Configuration Best Practices article, be sure to consult standard documentation such as the Getting Started Guide, User s Guide, Administrator s Guide, and MPX Administrator s Guide. 3

4 Performance and Scalability Techniques Architecture Overview 2 Architecture Overview The core components of client/server architecture are depicted in Figure 1 - /Server Architecture. s Server Process Server Configuration Win32 Database Repository Web Edition SDK Archive StarDisk SDK Server Cache Vault Attachments Custom App. SDK Figure 1 - /Server Architecture This diagram illustrates the typical components of a single deployment. From right-to-left, the major components in the architecture are described below. A single deployment instance of is known as a server configuration. All projects, views, change requests, and other artifacts that can be interrelated and otherwise managed as a whole reside in a single server configuration. Note that in this article, the term server configuration is sometimes shortened to simply configuration. With the exception of file contents, all data and metadata of a configuration are managed in a single database termed the repository. supports a variety of commercial database systems for the repository, allowing you to choose a database platform that best supports your organizations preferences. leverages available performance capabilities such as native database APIs, granular (row-level) locking, and ACID transactions. The terms repository and database are sometimes used interchangeably Historically, the term repository has been overloaded with several definitions, ranging from a synonym for configuration to a specific subset of persistent files. In this document, the term repository reflects a more common industry definition, equating it to the database. 4

5 Performance and Scalability Techniques Architecture Overview File contents are managed within operating system-level files collectively known as the vault. The vault is divided into three distinct areas that store specific types of content: Archive: The archive contains the base version and revision information for every file checked into the configuration. Revision information consists of forward deltas for text files and complete contents for binary files. When a versioned file is first added to a configuration, a new archive is started, which contains the base version of that file. As new versions of the file are created, revision information is added to that file s archive. Hence, an archive contains the complete revision history of a file, allowing any file revision to be reproduced. Cache: The cache contains pre-assembled file revisions as independent files, retained on a mostrecently-used (MRU) basis. When a new revision of a file is checked into a configuration, a copy is placed in the cache, providing quick access to that revision of the file. The cache saves the time required to construct historic revisions of popular files by applying deltas to the base versions of those files. Attachments: The vault also maintains an area for non-versioned, ancillary files known as attachments. The location where attachments are kept is configurable. Like archive and cache files, attachments reside outside of the repository database. Attachments do not typically possess significant size or performance implications 1. All information for a server configuration is stored in a single repository and corresponding vault, which must be kept in sync. When you backup a configuration, you must copy the database, archive folder, and attachments folder at the same time. However, you don t have to copy the cache since cache files are recreated automatically as needed from the archive. Each server configuration is managed by a single Server process. The Server performs all repository and vault management functions and responds to all client requests. It uses a multithreaded architecture, allowing it to take advantage of multi-cpu machines. The Server provides configurable caching options that can be used to improve the performance of various functions at the cost of additional memory use. A variety of s support a wide range of user roles. Some clients, such as the Win32 and StarDisk, are only available on Windows platforms. The SDK supports applications via Java and COM or.net-based APIs. Many SDK-based clients such as the Cross-Platform and Command Line Interface (not shown) are available on any platform that supports Java. Web Edition uses a browser client to provide a zero-install, web-based interface. Custom Applications that use the SDK can be developed by customers or Borland Professional Services to meet specific needs. All clients can connect to multiple server configurations simultaneously. Each server is accessed with a dedicated TCP/IP connection to the corresponding server process. Connection options include encryption for secure communication and compression for improved performance over slower network connections. A server can be configured to require encryption for all client connections or for those received by specific IP addresses. 1. Technically speaking, attachments are not part of the vault. However, because their storage, planning, and management issues are similar to those for the cache and archive, this article encourages you to think of attachments as just another vault component. 5

6 Performance and Scalability Techniques Hardware and Capacity Planning 3 Hardware and Capacity Planning This section provides information useful in the planning stages of a new deployment or the upgrade of an existing one. Basic options for deploying server components and the effects of various hardware resources on components are discussed. Guidelines are provided for estimating the size of vaults and repositories. Suggestions are provided for specific situations where choices can be made to improve performance. 3.1 Server Component Distribution Options In some cases, you can place server configurations on a single server machine. In other cases, you should distribute components of a single configuration across several machines. Scenarios for sharing and distributing server components across machines are discussed in this section Shared Server Machines For small- to medium-sized server configurations, you can place all server components on a single machine. You can also deploy multiple configurations on the same machine depending on the total size and total number of concurrent users of all configurations. This is illustrated in Figure 2 - Multiple Configurations on a Shared Machine. Shared Server Machine Server Configuration #1 Database Vault Configuration #2 Server Database Vault Figure 2 - Multiple Configurations on a Shared Machine When you deploy multiple configurations on the same machine, configure each server process to accept inbound connections on a different TCP/IP port. Also, be sure the vault component folders (cache, archive, and attachments) for each configuration do not overlap. You can configure each server configuration independently as a service, and you can perform administrative functions (lock, stop, backup, etc.) for each configuration independently. You should use a shared server machine only when 6

7 Performance and Scalability Techniques Hardware and Capacity Planning the total number of concurrent users is less than 100 or so, and the total number of projects is no more than Large Configurations Distributed Over Multiple Servers For a larger configuration or one in which you expect heavy concurrent access, you should run the Server process on its own machine with high-speed disks. Configurations with high concurrency can also benefit from deploying the database on a separate machine with high-speed disks. If you install MPX and expect high network usage, the MPX Message Broker can also be installed on its own machine to offload network traffic. Hence, a single server configuration can be served by up to three machines: one each for the Server process, database, and MPX Message Broker. An example of a single, large configuration taking advantage of multiple servers is illustrated in Figure 3 - Large Configuration Utilizing Multiple Servers. Server Machine Vault Database Machine Server High-speed network Database MPX Message Broker Message Broker Machine Figure 3 - Large Configuration Utilizing Multiple Servers When the database is deployed on a separate machine, the network interface between the server process and database machines should be high-speed (100Mb-1Gb). You should also consider using a dedicated network segment for this traffic Multiple Configurations Sharing a Database Server If you have multiple large server configurations, each Server process should run on its own machine. Multiple configurations can share the same database machine to simplify monitoring and administrative functions. Using a large machine (e.g., 4-CPU/4GB), some customers have deployed up to four large configurations sharing a single database machine. 7

8 Performance and Scalability Techniques Hardware and Capacity Planning A single MPX Message Broker can serve several Server processes, although a separate machine will offload outbound network traffic. The use of multiple MPX Message Brokers and the MPX Multicast Service are discussed later in the section MPX Server. An example of multiple configurations sharing database and Message Broker machines is shown in Figure 4 - Multiple Configurations with Shared Machines. Server Machine #1 Server Shared Database Machine MPX Message Broker High-speed network Database #1 Database #2 Server Server Machine #2 Figure 4 - Multiple Configurations with Shared Machines As in the previous example, high-speed networking should be used between the Server process and database machines. 3.2 Server Component Resource Factors The optimal hardware for your environment depends on many factors including how many server configurations you deploy, how many artifacts are stored in each, the number of concurrent users you expect, your organization s growth expectations, and so forth. Even when you have precise metrics for these factors, translating them into hardware requirements is an estimation process, not an exact science. This is due, in part, to the natural fluctuation in repository usage factors: file sizes, growth rate, how many users perform large check-outs at the same time, etc. Because of this variability, it is beneficial to understand general usage trends and how various components consume hardware based on usage. The following sections describe how server modules are affected by various hardware resource types. 8

9 Performance and Scalability Techniques Hardware and Capacity Planning Server Process Hardware Factors The resources most consumed by the Server process are summarized below in the order of importance:! Memory: The Server makes extensive use of caching. Some caching options are optional, but the more caching options are used, the greater performance the server provides in concurrent environments. The minimal recommended memory for the Server machine is 256MB. Server machines supporting more than users should have 512MB of memory. With concurrent users, the server should have 1-2GB of memory. Servers that support 200 concurrent users or more should have 2-4GB of memory. Note that on a 32-bit Windows OS, the maximum virtual memory that a process can use is 2GB. Memory above 2GB is reserved for drivers and other resources. Where memory is critical, the Server can leverage the Windows 4GT RAM Tuning feature to use up to 3GB of virtual memory. Contact Borland support for more information if you think this option is needed in your environment.! CPU: Due to its use of caching, the Server attempts to trade database and vault disk access for memory access where possible. This means that the server process is not necessarily disk access constrained and will benefit from increased processor speed, L1 and L2 cache levels, and other CPU performance factors. Furthermore, due to its multithreaded architecture, the server benefits from multiple CPUs. If you except concurrent users, consider a dual-cpu server machine. For users or more, consider a quad-cpu machine.! Network: In high concurrency environments, network bandwidth will have a significant impact on Server performance. If the database operates on a separate machine, the server should use a dedicated back side network segment to those machines using 100Mbit 1GBit Ethernet. The front side client network can become a bottleneck depending on how many clients are connected to the server and what operations they are performing. Most client transactions do not exchange a lot of data, but a few do, notably full scope operations and bulk file check-outs. A 100Mbit Ethernet will usually support up to concurrent users without network congestion. In higher concurrency environments, you should consider 1Gbit Ethernet or multiple NICs on the Server machine. Virtually all of the Server process s disk I/O will be to the database and vault, hence disk hardware issues are discussed under those components Database Hardware Factors Because the contents of your repositories are important business assets, you should treat your database similarly as other mission-critical databases. Your database administrator (DBA) should be familiar with the configuration, tuning, backup, and other administrative procedures for the database platform you use choose. Virtually all of the performance and scaling techniques you would normally use for a transaction-intensive database application apply to. When considering database hardware configuration, some guidelines generally apply to all database types. These are summarized below:! Memory: Like the Server, database systems can use memory caching to improve performance. For large configurations, this argues for placing the database on a separate machine, installing as much memory as possible, and configuring the database to make use of it.! Disk: Fast disks and disk striping can have a significant affect on database I/O performance. Because some RAID systems provide both improved performance and fault tolerance benefits, 9

10 Performance and Scalability Techniques Hardware and Capacity Planning you should consider a RAID system for your database. An overview of RAID systems is provided in Appendix - RAID Systems.! CPU: Like the Server, database systems are multithreaded and will benefit from faster CPU speeds and multi-cpu systems. This also argues for placing the database on a separate, multi-cpu machine.! Network: As previously discussed, then the database is deployed on a separate machine, highspeed networking to the Server process machines potentially using a dedicated network segment is important. Database capacity planning and performance tuning techniques are discussed in later sections Vault Hardware Factors The vault is used for all file-based activity: check-in, check-out, and attachment operations. In terms of I/O activity, the demand on each vault component is summarized below:! Cache: The cache is the heaviest used vault component. New files are added to the cache at check-in; files already in the cache are used for check-out; and non-cached files are added to the cache at check-out.! Archive: The archive is the next most heavily used vault component. An archive file is added or updated for each file check-in, and an archive file is accessed when a check-out of a noncached file occurs.! Attachments: Attachment files generally experience light usage. Files are added to or read from the attachments folder only when accessed via CR, task, topic, and requirement objects. Ideally, vault components should be spread over multiple, high-speed disks attached to the Server process machine. Your first choice should be to separate the cache and archive. If you can, separate both of these from the system disk where the operating system, page file, and other operational files are located. If you deploy the database on the same machine, keep the database partitions and vault components on separate disks. If you have enough disks, put the database, cache, archive, and system components on separate disks to maximum I/O concurrency. If a RAID system is used for the vault, both the cache and archive components can benefit from performance features. However, only the archive component benefits from fault tolerance/redundancy features since the cache is completely redundant with information derivable from the archive. Attachment files can benefit from fault tolerant features but do not generally require RAID performance features. In general, placing the vault on a separate networked machine is not beneficial. Extending archive and cache I/Os over a network is usually slower except when specialized, high speed network storage devices (e.g., network appliances ) are used. These devices are usually costly and are beyond the scope of this article MPX Message Broker Hardware Factors Even when the Message Broker is deployed on the same machine as a Server process, it reduces overall network demand by pushing update events to clients, preventing them from repeatedly pulling content using polling and refresh techniques. Hence, deploying MPX always results in a net increase in client responsiveness and server throughput even when no additional hardware is used. Since it is as a separate process, the MPX Message Broker represents an opportunity to offload even more network traffic from the Server process machine by placing it on a separate machine. 10

11 Performance and Scalability Techniques Hardware and Capacity Planning However, this server does not need to be a high end machine. The Message Broker does not generally benefit from increased memory or multiple CPUs. This is because the Message Broker is primarily a communication server and consumes proportionately little CPU and memory. Disk-wise, the Message Broker performs virtually no disk I/O Usage Performance Factors The usage patterns with which configurations are used can significantly affect the performance of certain operations. Some of the factors that have the largest impact on memory usage, database performance, or overall client responsiveness are summarized below.! Concurrent clients: Each concurrently connected client consumes a small amount of resources within the Server process. Each active client request consumes an available worker thread and potentially an available database connection. (Database connections are pooled and shared among worker threads.)! Open views: Each view opened by at least one client utilizes memory for caching information associated with that view. A view that has been temporarily rolled back in time for a client session requires similar caching as a current view.! Number of items per view: The amount of cache memory required for an active view is proportional to the number of items visible within that view.! Number and depth of branched views: As the number or depth of branched views increases, certain transactions consume proportionately more database processing.! Derived view creation: The time required to create a derived view is proportional to the number of visible artifacts (view members) within the parent view. 3.3 Database Capacity Planning supports Oracle, Microsoft SQL Server (and MSDE), and IBM DB2 database systems for its repository. A repository that supports a small team (up to users) and a small to moderate number of artifacts (up to several thousand) can utilize MSDE as the repository database. Repositories that support large teams (dozens to many hundreds of users) and/or a large number of artifacts (tens of thousands or more) should use one of the other commercial database choices. Initially, a new repository database contains 1-2MB of data, consisting of the catalog. (Note that this figure is the data size and does not include block and partition rounding, which vary by database.) After initialization, the size of the data grows proportional to the number of artifacts placed within the repository and the number of revisions made. For most repository object types (users, groups, security objects, etc.), the space consumed in the underlying database is not significant either because the corresponding objects are small or because they constitute a proportionately small percentage of the database. The majority of growth within repositories occurs in three types of information: Object Tables: For each type of versioned object (files, folders, change requests, tasks, and topics), a corresponding table is used to store that artifact type s data. One record is created for the initial version of each object, and an additional record is created for every subsequent revision of the object. Object records typically consume between bytes each. Additionally, each artifact table is spanned by 2-4 indexes, which adds ~50% more data per record. For a round estimate, assume 400 bytes per object revision. View Member (Item) Table: Each time a versioned object is included in a view, a view member or item is created. Items both associate the object with the view and store view-specific properties for the object. All item data is stored in a single table, which is spanned by multiple 11

12 Performance and Scalability Techniques Hardware and Capacity Planning indexes. Together, the table and index records require ~300 bytes of space per item. Note that when objects are shared among multiple views (e.g., when a child view is derived from a parent view), items are created for each view. For example, if 1,000 files are visible to three different views, 3,000 items (requiring ~900KB) are created. Log Tables: Within the repository database, maintains two types of log tables that record specific event types. The Audit table is a journal-like history of all changes made to versioned objects. The Security Log table records security-related events including artifact-related changes (e.g., access right changes) and session-related events (e.g., failed logons). In both cases, each table record is relatively small, but both tables will continually grow to the corresponding configuration limits. The rate at which these tables grow is dependent upon the occurrence rate of the corresponding events. To the degree that the number of records and the rate of change in each of these three information types can be estimated, the size and growth rate of a configuration can be projected. 3.4 Vault Capacity Planning As previously described, each configuration contains a vault, consisting of archive, cache, and attachment folders, which can be separately located. For new configurations, all three folders are empty. As file objects are added to the configuration, vault files are added to the cache, archive, and attachment folders. Factors that affect the growth of the archive are summarized below:! When a new versioned file is added to a configuration, a new corresponding archive file is started on its behalf.! The archive file contains the versioned file s initial contents plus a small amount of meta information. By default, this information is compressed, so the initial archive file is typically smaller than the original file. Compression ratios vary depending on data, ranging from virtually no compression with already-compressed file formats (e.g.,.zip and.gif) to nearly 5:1 for text files with lots of white space or repeating content. The following table illustrates typical original-to-archive file size ratios using default compression: File Type Original Size Initial Archive File Size Ratio Text (HTML) 29KB 10KB.36 Binary (PDF) 248KB 186KB.75 Source (C++) 33KB 8KB.24! The compression used for a specific file s archive can be configured. Default compression provides a good balance of effective yet fast compression. Maximum Speed, Maximum Compression, and None are available as alternate compression techniques.! As new revisions of a versioned file are checked in, revision information is appended to the corresponding archive file. If the versioned file is text-based, the revision information is by default stored as forward deltas and is proportional to the size of the change. For binary files, the complete new file revision is by default appended to the archive file. In both cases, the information is compressed at about the same ratio as the initial file contents. This behavior can be used to estimate the size requirements of the archive for a given file. 12

13 Performance and Scalability Techniques Hardware and Capacity Planning! Note that for very large files, the archive compression technique affects the maximum number of revisions that can be stored for the file. This is because the size of an archive file is currently limited to 4GB (for 5.3). Factors that affect the size of the cache are summarized below:! When a new file or a new revision of an existing file is added to a configuration, a copy of the file is placed in the cache. This makes the file available for immediate access to subsequent check-out operations (since check-outs are usually requested soon after new file revisions are made.)! When the check-out of a tip or historic version of a file is requested and that file is not available within the cache, the requested file version is constructed from its archive. A copy of the fully constructed file is then added to the cache.! A most-recently-used (MRU) list is maintained for all files within the cache. When a new file is added to the cache or when an existing cache file is requested, the file is moved to the top of the MRU list.! The total size of all files within the cache is monitored by a Server background thread. When this size exceeds the configured maximum cache size, the least recently used files (at the end of the MRU list) are deleted from the cache.! The maximum size of the vault cache (for 5.4) is 4GB.! Due to its performance benefits, the cache should be configured to be large enough to hold the tip revision of all versioned files. Attachment files are stored in the attachment folder without compression or delta storage. Since attachments are un-versioned files that provide ancillary information to CRs or other objects, the total size and growth rate of these files is usually very small compared to the archive size. A practical approach for managing attachments capacity is to focus on archive capacity requirements and simply place attachments on the same disk. 3.5 Server Component Hardware Recommendations To round out hardware and capacity planning, this section provides some example hardware recommendations for various configurations. The examples below suggest minimum and recommended hardware for the server on which the Server process and database are deployed. Note that, although specific processor speeds are provided, you should always seek the latest (i.e., fastest) CPU speeds to maximum performance and MSDE on the Same Machine When MSDE is used, the database typically resides on the same machine as the corresponding server process. Recommendations for a joint server/msde machine are given below. 13

14 Performance and Scalability Techniques Hardware and Capacity Planning Number of Seats (registered users) Minimal Configuration Recommended Configuration < 50 Pentium 4, 1.3 GHz, 1.5 GB of RAM Dual Pentium Xeon, 2.26 GHz+, 2.5 GB of RAM Dual Pentium 4, 1.3 GHz+, 2 GB of RAM Same as minimal It is not recommended to use MSDE for configurations using more than 100 registered users Server Alone The following guidelines apply when the Server process is executing on a machine by itself. Number of Seats (registered users) Minimal Configuration Recommended Configuration < 50 Pentium 4, 1.3 GHz, 512 MB of RAM Dual Pentium 4, 1.3 GHz+, 1 GB of RAM Dual Pentium 4, 1.3 GHz, 1 GB of RAM Dual Pentium Xeon 4, 2.26 GHz and 1.5 GB of RAM. Dual Pentium 4, 1.3 GHz+, 2 GB of RAM Dual/Quad Pentium Xeon 4, 2.26 GHz and 2.5GB of RAM. Over 200 Any high performance Enterprise Server with RAID system (recommended), quad processors with 2.26 GHz+, 4.0 GB of RAM Same as minimal 14

15 Performance and Scalability Techniques Hardware and Capacity Planning Database Server Hardware Recommendations The following recommendations apply to the database server when it is not shared with the server process: Number of Seats (registered users) Hardware Configuration Database Requirements < 50 Pentium 4, 1.3 GHz, 1 GB of RAM Minimum: MSDE 2000 SP2 Recommended: Any Enterprise level database such as SQL Server, Oracle, or DB Dual Pentium 4, 1.3 GHz, 2 GB of RAM Dual/Quad Pentium Xeon 4, 2.26 GHz and 2-3GB of RAM Minimum: MSDE 2000 SP2 Recommended: Any Enterprise level database such as SQL Server, Oracle, or DB2 Recommended: Any Enterprise level database such as SQL Server, Oracle, or DB2 Over 200 Any high performance Enterprise Server with RAID system (recommended), quad processors with 2.26 GHz+, 4.0 GB of RAM Recommended: Any Enterprise level database such as SQL Server, Oracle, or DB2 15

16 Performance and Scalability Techniques MPX Server 4 MPX Server MPX Server is packaged with Enterprise Advantage and available as an add-on product for Enterprise and Standard editions. For even moderate-sized development groups, MPX is an excellent enhancement to boost overall performance and server scalability. MPX Server, which is also referred to as simply MPX, increases the scalability of Servers and improves the responsiveness of clients via advanced caching techniques. An overview of the MPX Server architecture is shown in Figure 5 - Overview of MPX Server Architecture. MPX-enabled clients Non-MPX clients MPX-enabled clients client/ server requests event messages MPX Message Broker Database Server broker-to-broker forwarding event API MPX Event Xmitter encrypted event messages MPX Message Broker Figure 5 - Overview of MPX Server Architecture The first component added by the MPX Server architecture is the MPX Event Transmitter. This module uses an event API to receive asynchronous notifications on server-related events. The Event Transmitter converts these events into encrypted XML messages and forwards them to the MPX Message Broker. The Message Broker provides a unicast publish/subscribe communication and messaging service, broadcasting event messages to clients with updates specific to their current sessions. (An alternate multicast deliver method is also available, which is discussed later.) MPX enhances the scalability and performance of environments in a number of ways:! MPX reduces the number of client/server requests between clients and the Server. By pushing new and updated information to clients rather than waiting for clients to pull it, fewer client/server requests are made upon the Server. Broadcasting an update event once requires less server processing and network traffic that polling for updates repeatedly by many clients. As a result, resources are freed for more users and faster responses.! Because clients receive updates immediately, many operations are more responsive since fewer server round trips are required. In the case of client refresh (F5) and check for new and modified items operations, no client/server requests are issued when running in MPX 16

17 Performance and Scalability Techniques MPX Server mode. Also, in MPX mode, clients provide an instant refresh option that automatically refreshes the display when new information is received.! The MPX Message Broker can operate on a separate machine to offload demand from the Server. From the Server, each event notification is given once, asynchronously, to the event transmitter, which publishes a single message to a Message Broker. The Message Broker takes responsibility for delivering notifications to each client on a subscription basis. In high volume environments, the Message Broker can be operated on a machine independent of the Server, freeing machine resources to support more users.! In geographically dispersed environments, multiple MPX Message Brokers can be deployed and connected into message communication clouds. clients and MPX Event Transmitters can connect into the cloud via their closest Message Broker. Message Brokers automatically use broker-to-broker forwarding to route each event message once across long distances, where it can be distributed locally to clients. With the current release of MPX Server, refresh and other operations normally sent from Win32 clients are typically reduced by 36%. The freed Server resources can consequently service a larger number of clients without increasing hardware. Some scenarios for deploying MPX are discussed next. 4.1 Use MPX for Distributed Teams MPX can help improve the responsiveness of distributed teams by reducing client/server requests over long distances. There are several variations of this scenario including the following:! You have developers who work from home or small remote offices using the public Internet to access your server. External developers network bandwidth is limited by their ISP, maybe 1Mb for DSL or cable and 56Kb for dial-up, compared to 100Mb+ that their on-site peers enjoy.! You have a remote development team that connects to your server over a VPN, but since all corporate traffic flows over the VPN, bandwidth is precious.! You have mobile developers who connect to your server remotely from customer sites, hotel rooms, and Internet cafés, experiencing a wide range of connection speeds, but requiring more than just casual access to your production server. In each of these scenarios, MPX can improve the responsiveness of remote developers. First, take the case of remote users connecting from the Internet. You can deploy a Message Broker outside the firewall, or you can open a port to your internal Message Broker for external access. In both cases, the Message Broker has no persistent data and its messages are encrypted with a dynamic key, so public access is safe. In Figure 6 - Access to Message Broker via Firewall, both the Server process and Message Broker are shown available to external clients by appropriate configuration of the firewall. With network address translation (NAT), both services can be given a unique IP address and/or port number, causing connections to be directed to the internal IP address and port number used by internal clients. 17

18 Performance and Scalability Techniques MPX Server external clients Firewall external Server port external Message Broker port Server MPX Message Broker internal clients Figure 6 - Access to Message Broker via Firewall Another strategy is to give the Server and Message Broker machines dual NICs, connecting one NIC to the internal segment and one to the external network segment. If corporate policy requires a restricted number of ports available to the public Internet, both the Server and Message Broker can be configured to use a well-known port (e.g., 80) as long as no other process uses the same port on the same machine. To support a remote team larger than 5-10 people, a secondary Message Broker can be deployed, chained to the primary Message Broker. Each team uses the Message Broker closest to them, minimizing both command traffic and MPX event traffic over the WAN connection. This scenario is shown in Figure 7 - Secondary Message Broker for Remote Team, with a remote development team using a VPN. 18

19 Performance and Scalability Techniques MPX Server local clients remote clients client/server requests Server VPN MPX Message Broker broker-to-broker forwarding MPX Message Broker event messages event messages Figure 7 - Secondary Message Broker for Remote Team Up to 10 Message Brokers can be connected in a single message cloud. To instruct a Message Broker to establish a connection with another Message Broker, add the following line to its STMessageBroker.ini file: setopt server_names tcp:mbserver1:5101, tcp:mbserver2:5101 where mbserver1, mbserver2, etc. are the host names or IP addresses of each remote Message Broker server, and 5101 is the appropriate port number with which that Message Broker is receiving inbound connections. It is OK for two Message Broker to redundantly point to each other, but you should avoid unnecessary Message Broker connections. For most organizations, Message Brokers should use a hub and spoke configuration, with each remote Message Broker connecting to the central Message Broker only. 4.2 Use Multicast for Large LANs Message Brokers broadcast messages to their clients using unicast messaging. One advantage of unicast messaging is that it operates along any path that allows TCP/IP: across routers, through firewalls, etc. Although each message is sent separately to each client, the overall network traffic afforded by unicast messaging is still lower than client/server request traffic without MPX. However, MPX network traffic can be reduced even further by using multicast messaging. To use multicast messaging, another MPX component termed the MPX Multicast Service is deployed. The Multicast Service is optional and separately installed. The Multicast Service must be paired with a single Message Broker, and it is normally installed on the same machine as the Message Broker. The Multicast Service broadcasts messages to MPX clients using IP multicast, which allows a single message to reach all clients. In effect, the broadcast network traffic remains constant regardless of the number of subscribing clients. The result is a further reduction in overall network traffic. 19

20 Performance and Scalability Techniques MPX Server An example deployment of the Multicast Service is shown in Figure 8 - Example MPX Multicast Service Usage. As shown, multiple Multicast Services can be deployed, each serving a local community of users. Also, some clients can use unicast message (via a Message Broker) while other clients can us multicast messaging (via a Multicast Service) at the same time both types of clients receive the same messages. Server site unicast clients remote site unicast clients Server unicast event messages MPX Message Broker VPN, WAN, etc. MPX Message Broker MPX Multicast Service MPX Multicast Service multicast clients multicast event messages multicast clients Figure 8 - Example MPX Multicast Service Usage If you decide that multicast messaging may be advantageous in your environment, deploy the Multicast Service following these guidelines:! Multicast only provides meaningful network traffic savings for larger user communities (50 or more clients connected to the same Multicast Service).! Multicast should only serve clients on the immediate LAN. This is because IP multicast traffic is usually blocked by firewalls and some VPNs. Also, network routers are sometimes configured to block multicast traffic. So, if you need multicast traffic to traverse a router, ensure that multicast forwarding has been enabled.! It is not necessary (or advisable) to deploy multiple Multicast Services in the same local area. One Multicast Service per geographic area is sufficient.! Although clients receive event messages via IP multicast, they must also establish a TCP/IP connection with the Multicast Service, which is used for channel negotiation and control traffic. This TCP/IP port (5104, by default) is configurable. If you deploy a Multicast Service or more than one Message Broker, you ll want to use MPX profiles, which is discussed next. 20

21 Performance and Scalability Techniques MPX Server 4.3 Use MPX Profiles for Multiple Messaging Services The ability to deploy multiple Message Brokers and Multicast Services gives you many options for using MPX in your environment. However, it also requires that each client use the Message Broker or Multicast Service most appropriate to them. To simplify the management and selection of the appropriate messaging service, provides a facility known as MPX profiles. Simply put, an MPX profile defines the information needed to use a specific Message Broker or Multicast Service. This includes connection information (host name/ip address and port number) and usage properties such as timeout and keep-alive values. All profiles are kept on the Server machine in an XML file specific to a server configuration, and the profiles can be managed via the Server Administration tool. The Server Administration allows you to copy, edit, add, and delete profiles. There are two kinds of profiles: A unicast profile defines a connection to a Message Broker, while a multicast profile defines a Multicast Service profile. A template profile XML file is created when you install MPX, containing both a sample unicast and multicast profile, even if you don t install multicast. This template is copied for each server configuration you create, after which you can edit and tailor it. Within the profiles for every configuration, you must make two designations:! Default Server Profile: This profile is the one used by the MPX Event Transmitter. It should always be a unicast profile since the Event Transmitter is a pure publisher and receives no benefit by using a Multicast Service. Often, the default server profile will define a Message Broker installed on the same machine as the Server process.! Default Profile: This profile is the one that will be used, by default, by all MPX-enabled clients that have not chosen an alternate profile. The same profile can be both the server and client default profile; in fact, this is probably what you want if you re not using multicast. The default client profile should specify the Message Broker or Multicast Service that the majority of clients should use. s that should use a messaging service other than the default must, on a one time basis, choose the appropriate alternate profile. In the Win32 client, this done by clicking MPX Profiles on the appropriate Server Properties dialog and then selecting the alternate profile. The selection is saved at the client and subsequently used for all future connections until it is changed. 21

22 Performance and Scalability Techniques Performance Tuning 5 Performance Tuning Regardless of the underlying hardware or the component distribution architecture you use, there are many options for tuning configuration performance that you apply. The most useful tips and techniques are described below. 5.1 Server Tuning Techniques These options affect the operation of the Server process StarbaseServer.ini File Options Global options for each server configuration are defined in the StarbaseServer.ini file. Options that can be tuned for on a per-configuration basis are described below Object Caching The caching options FilesCaching, ChangeRequestsCaching, TasksCaching, and TopicsCaching each allow an integer value from 0 to 2. The value 0 indicates that properties for the corresponding objects (files, CRs, tasks, or topics) are not cached. A value of 1 indicates that properties of tip (latest) revision objects are cached when they are first touched. A value of 2 indicates that tip objects are pre-cached at server startup time, providing immediate performance benefits even to the first access of the corresponding objects. By default, FilesCaching is set to 2. File objects use comparatively little memory, hence precaching of tip file objects provides a good trade-off between memory usage and accelerated to access file objects. However, CR, task and topic objects use considerably more memory per object due to the larger number of properties, especially text properties. Consequently, the caching option for these objects is set to 0 by default. To improve overall access to any of these object types, you can set the corresponding caching option to 1 or 2, but balance the use of this option with the server process s available memory and the number of tip objects in your repository for the corresponding object type. Memory used by the caching options is not released by the server until the server is shutdown and restarted Compression and Delta Storage As new versioned files are added to a configuration, the use of compression and delta storage is set for each file. Compression saves archive disk space at a cost of CPU time when the file is first stored and subsequently accessed from the archive. (Files stored in the cache are not compressed, hence a check-out of a cached file does not access the archive.) Delta storage also saves disk space and allows delta checkouts, also at a CPU cost. By default, chooses compression for all new files and delta storage for new text files. You can override these settings per-file by setting the appropriate options when the file is added. For large files, however, the default values may not be optimal. For example, delta storage may have little value for large XML and RTF files, even though they are text files. Similarly, some large binary files such as ZIP and GIF do not compress well, hence using compression storage in the vault is not beneficial. 22