VERITAS SANPoint Foundation and Suite SANPoint Foundation

Transcription

1 VERITS SNPoint Foundation and Suite SNPoint Foundation H Suite New VERITS Volume Management and File System Technology for Cluster Environments V E R I T S W H I T E P P E R September 2001

2 Table of Contents I. Clusters Technology for Continuous Computing Cluster Technology Technologies That Enable Clustering Online Storage for Clusters Data ccess Models for Clusters II. VERITS SNPoint Foundation Suite and VERITS SNPoint Foundation Suite H VERITS SNPoint Foundation Suite and VERITS SNPoint Foundation Suite H Components pplications for SNPoint Foundation Suite and SNPoint Foundation Suite H III. VERITS Cluster Volume VERITS Volume VERITS Cluster Volume IV. VERITS Cluster File System File System for Clusters VERITS Cluster File System enefits VERITS Cluster File System rchitecture V. Summary Why a VERITS Solution? Figures Figure 1: Cluster requirements Figure 2: Volume aggregation of enterprise RID subsystems Figure 3: Shared-nothing cluster model Figure 4: Shared-data cluster model Figure 5: Highly available file server Figure 6: Scalable Web server Figure 7: Video editing workflow application Figure 8: VERITS Volume volume architecture Figure 9: VERITS Volume volumes in a VERITS Cluster cluster Figure 10: Private and cluster-shareable disk groups Figure 11: VERITS Cluster File System layered on Cluster Volume Figure 12: VERITS Cluster File System server-client architecture Figure 13: VERITS Cluster File System integrated components Tables Table 1: Volume volume types

3 I. Clusters Computing for the 21st Century Technology for Continuous Computing With the rise of the Internet, and particularly the e-business explosion, continuous computing has become an operational necessity. Customers, suppliers and employees have become accustomed to accessing corporate information resources at their convenience. Downtime, whether it s because of hardware or software failure, because data must be backed up, or because a system upgrade is necessary, is no longer an option. Information has to be available all the time. IT managers are challenged to: Provide continuous application service and data access, even if applications or systems fail or data centers become incapacitated Provide secure universal access to enterprise data Provide application performance that grows as the organization grows and its computing requirements increase Contain management cost as the number and complexity of systems grows Cluster Technology Cluster technology helps meet these challenges by coordinating interconnected servers to enhance application and data availability, scaling potential and manageability. Client network Common client access Common data access Common storage interconnect (e.g., fibre channel hub or switch) Figure 1: Cluster requirements New VERITS Volume Management and File System Technology for Cluster Environments Page 1

4 s Figure 1 illustrates, a cluster is a set of interconnected servers (called nodes ) that are: Connected to the same storage ccessible by the same clients Operationally coordinated by a cluster server such as VERITS Cluster Clustered servers enhance application availability and application scaling potential. Some cluster servers, including VERITS Cluster, add a single-system image that supports management of all systems from one console. Clusters are conceptually attractive to users because of their potential to solve some of the most significant problems in computing: failure: If a server or application crashes, another server in the cluster can fail over its workload, providing nearly continuous service to clients. pplication growth: If application demands become too great for the existing servers, additional servers can be added and the workload repartitioned. Network failure: If network links fail, clients can continue to access applications on alternate paths. Disaster recovery: If an entire data center is incapacitated, cluster servers in other buildings can restart the workload and continue processing. Management cost: s an enterprise s applications grow in number, clusters can help contain management cost by minimizing the number of individual systems that must be managed. Technologies That Enable Clustering In order to form a cluster, servers must have common connections to clients and to data. Over the past decade, Ethernet-based local area networking has matured as the technology of choice for flexibly interconnecting large numbers of computers. More recently, fibre channel-based storage area networks (SNs) have enabled the connection of terabytes of storage to dozens of computers at high performance levels. oth Ethernet and fibre channel are standards-based technologies that are widely available and interoperable among computing platforms. Hardware interconnection technologies to enable clustering are a problem that has been solved. Less obvious from Figure 1 are the software technologies that are also necessary for cluster computing. Clustered servers that provide application failover must coordinate their actions so that applications fail over when and only when failover is appropriate. Clustered servers that balance load must coordinate their actions to distribute work equitably. Moreover, they must synchronize their accesses to data to avoid interfering with each other s updates. In order for clustered servers to deliver their potential benefits, they must have software that: Intercommunicates among the servers to provide them all with the same view of cluster state at all times Monitors application state so that unnecessary failovers are minimized and necessary failovers occur as quickly as possible Manages access to online storage so that different applications concurrent data accesses do not interfere with each other Cluster servers consist of the software that provides these services. VERITS Cluster implements all of these services in clusters of up to 32 interconnected Solaris, HP-UX, Windows NT or Windows 2000 servers. Page 2 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

5 Online Storage for Clusters Clusters are highly available, scalable, manageable computing environments. Ideally, these properties would be coupled with online storage that is also highly available, scalable and administrator-friendly. n ideal storage device for large clusters is an enterprise RID subsystem. Enterprise RID subsystems are engineered for high capacity and multiple host connections. Most tolerate disk, power supply, cooling device, host interface and cache failures without losing data or interrupting I/O service. Most are capable of storage, cache and host interface expansion, and include management tools that allow multiple subsystems to be managed from a single location. File system Large high-performance failure-tolerant volume Host-based volume manager Failure-tolerant virtual disk VERITS Volume stripes or mirrors data across virtual disks Failure-tolerant virtual disk Mirrored or RID arrays Enterprise RID subsystem Mirrored or RID arrays Enterprise RID subsystem Figure 2: VERITS Volume aggregation of enterprise RID subsystems Host-based logical volume managers such as VERITS Volume provide similar benefits at lower cost for smaller systems. In larger systems, volume managers complement enterprise RID subsystems either by combining RID arrays for greater availability and I/O performance, or by subdividing them to simplify the administration of very large amounts of storage. New VERITS Volume Management and File System Technology for Cluster Environments Page 3

6 Data ccess Models for Clusters Figure 1 suggests full interconnection of all servers and all storage devices in a cluster. Clustered servers must cooperate so as not to interfere with each other s accesses to file system metadata or user data. There are two basic cluster data access models: Shared nothing: With this model, each storage device is owned (accessed) by only one server at a time. Storage device ownership may pass from server to server, but a server must relinquish ownership before another can claim a device. In the shared nothing cluster model, applications running on different servers cannot access the same file systems concurrently. Shared data: With this model, storage devices can be accessed by more than one server at the same time. In the simplest variation of the shared-data model, servers share access to storage devices on which they create private logical volumes. more sophisticated shared-data model, implemented by the VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H products, supports concurrent access to file system data by all servers in a cluster. may not access storage owned by server may not access storage owned by server Common physical interconnect Volume owned by server Volume owned by server Figure 3: Shared-nothing cluster model Page 4 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

7 Shared-Nothing Clusters Shared-nothing clusters enhance application availability. If an application or the server on which it is executing fails, a failover server takes control of the application s storage devices, and restarts it. Shared-nothing clusters also enable read-only applications to scale beyond the capacity of a single server. Prior to the Internet explosion, read-only applications were of limited utility. Commercial Web servers, however, are often heavily loaded with read-only accesses to data. Multiple instances of a read-only Web application can run on clustered servers, each accessing its own copy of served Web pages. Clients may be routed to a particular server by a load balancing front end such as Cisco s Local Director or Microsoft s Windows Load alancing. s long as access is read-only, there is no need to synchronize copies of the Web pages. Shared-Data Clusters Shared-data clusters also enhance application availability, and, in addition, enable any partitionable application to scale beyond the capacity of a single server. Shared-data clusters provide read-write access to a single copy of data to multiple application instances executing on different servers. Since all applications access the same copy, all applications have instant access to all data updates. s coordinate their access to volumes or file systems so that updates do not interfere with one another Shared Volume Shared File System Figure 4: Shared-data cluster model New VERITS Volume Management and File System Technology for Cluster Environments Page 5

8 Enablers for Shared-Data Clusters Shared-data clusters are more broadly applicable than shared-nothing clusters. Sharing data, however, requires that application accesses be coordinated so that file system metadata updates from different servers do not overwrite each other. VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H integrate several VERITS technologies to provide this coordination, and enable concurrent sharing of volumes and file systems by clustered servers. pplication instances running on different servers that are able to coordinate their accesses to data, for example, by using application level locking, can even share access to data within the same file. Page 6 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

9 II. VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H Shared-Data ccess for Clustered Computing VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H Components VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H enable coordinated access to shared data in clusters. Each product integrates VERITS Foundation technologies: VERITS Volume VERITS File System with intercommunication technologies from VERITS Cluster : the Global tomic roadcast (G) cluster membership protocol the Low Latency Transport (LLT) inter-node messaging protocol and adds unique technologies of its own: VERITS Cluster Volume VERITS Cluster File System to provide shared volume and shared file system access to the nodes of a cluster. pplications for SNPoint Foundation Suite and SNPoint Foundation Suite H VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H provide a robust shared-data access capability that enables or improves a variety of shared-data cluster applications: For multi-instance applications, such as distributed database management systems and Web applications, that run on several servers concurrently, SNPoint Foundation Suite and SNPoint Foundation Suite H make a single robust file system available to all application instances. SNPoint Foundation Suite and SNPoint Foundation Suite H allow these applications to grow by adding servers, and also improves availability by enabling them to redistribute load when a server fails simply by reassigning network addresses. For workflow applications such as video production, in which large files are passed among workstations, SNPoint Foundation Suite and SNPoint Foundation Suite H eliminate time consuming and error-prone data copying by making files available at all stations. For backup, SNPoint Foundation Suite and SNPoint Foundation Suite H reduce operational impact by allowing backup to run on a separate server, reading data from shared file systems. For single-host or multiple-host applications that require continuous availability, SNPoint Foundation Suite H reduces application failover time by providing a running file system with which an application can restart after a server failure with the application failover provided by VERITS Cluster. The following describes how SNPoint Foundation Suite and SNPoint Foundation Suite H enhance the quality of services that these applications can deliver. New VERITS Volume Management and File System Technology for Cluster Environments Page 7

10 Continuous vailability: Enterprise File Services Clustering is popular for enhancing file server availability. Two or more servers in a cluster configuration (connected to the same clients and the same storage) serve separate file systems. If one server fails, the other recognizes the failure, recovers and mounts its file systems, and begins responding to clients. The left side of Figure 5 illustrates this model. File service /admin /data failure requires restart of file service(s) on alternate servers File service /humres File service /admin /data /humres VERITS SNPoint Foundation Suite failure does not require a file service restart on other server(s) VERITS SNPoint Foundation Suite File service /admin /data /humres /admin /data /humres /admin /data /humres Discrete file systems SNPoint Foundation shared file systems Figure 5: Highly available file server While certainly more available than single-host file serving, this model is somewhat inflexible, in that each file system is bound to one server. Responding to environmental changes, for example, an increase in the capacity required by one file system and a decrease in that of another, consumes administrative time and talent. Contrast this with clustered file serving using SNPoint Foundation Suite and SNPoint Foundation Suite H, illustrated on the right side of Figure 5. ll file systems are accessible by both servers, so either can serve any file. Clients might choose a server based on loading or other considerations, or the cluster itself might balance load by directing client requests made to a cluster IP address to the least busy server. This configuration offers other advantages as well: dministration is simpler there is never a question of which server should have storage added, or which server should serve additional file systems as they are added Service availability (uptime) improves, since a server crash or shutdown does not interrupt file service, nor does it require file systems to be restarted or remounted on other servers (SNPoint Foundation Suite H with full VERITS Cluster is required for application failover) vailability of critical data may improve if the pooled storage resources enable more file systems to be placed on mirrored volumes Performance may improve due to I/O load balancing if the volumes holding the file systems are striped across more disks Page 8 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

11 Parallel pplication: Web Web serving is one of today s highest growth applications. Web serving is well-suited for shared-nothing clusters because data access is often read mostly, so little data synchronization is required. client load-balancing facility (e.g., Cisco s Local Director or Microsoft s Windows Load alancing ) can be added to optimize server resource utilization. Web serving cluster s capacity can be expanded simply by adding a server and another copy of data. In this scenario, illustrated in Figure 6, each server has its own copy of Web data. If a server fails, it ceases to be part of the client load balancing; the workload is distributed among the remaining servers. No coordination of Web server data Client Load alancing Service (may also be provided by network) Load alancing Service Web server for site X Site X Each Web server has a separate copy of data Site X Load alancing Service Web server for site X Load alancing Service Web server for site X ll servers are guaranteed to serve the same data Site X Load alancing Service Web server for site X Site X SNPoint Foundation shared file system Figure 6: Scalable Web server This model is popular, and some of the world s largest Web sites are based on it. The shortcomings are again primarily administrative. When site content changes, the multiple copies of data must be updated carefully so that clients do not experience different behavior depending on which server handles a request. Significant administrative effort is required to manage clusterwide updates. In most cases, the financial cost of incremental storage and administration and the difficulty of maintaining the identity of many copies of data ultimately become prohibitive. shared-data cluster based on SNPoint Foundation Suite and SNPoint Foundation Suite H greatly simplifies scaling and administration for this type of application. s the right side of Figure 6 illustrates, a shared-data cluster enables all Web servers to work from one data image. Clients are guaranteed to see the same data, no matter which server handles a request. ll administrative effort and computing resources required to maintain multiple data copies are eliminated. When capacity needs grow, servers can be added to meet increased client access demands, or storage can be added to meet increased data or I/O performance demands, each independently of the other. Since these additions do not require restructuring of the operation (e.g., no file systems or private storage subsystems are added), adding capacity is simpler than with a shared-nothing cluster. New VERITS Volume Management and File System Technology for Cluster Environments Page 9

12 Workflow pplication: Video Production Video post-production is the archetype of a workflow application, in which a single piece of work is passed from computer to computer as it makes the transition from raw material to finished product. finished video clip starts as one or more raw digitized images. series of specialists edit content, scene transitions, light and color, and titles. Each specialist takes as input the output of the previous specialist in the chain, uses a high-performance workstation to perform his function, and passes the output to the next specialist. Video clips may be multiple gigabytes in size. Moving them from one workstation to another either by network or tape, as illustrated on the left side of Figure 7, is time-consuming and error-prone. Message Message Network or tape transfer Image Network or tape transfer Clip C Image Clip C Data capture app Editing app Produce final product Data capture app Editing app Produce final product Raw video Edited clips Final product Separate file system for each stage Edited clips Raw video Final product SNPoint Foundation shared file system Figure 7: Video editing workflow application The left side of Figure 7 illustrates workstations with private storage. When workstations are connected to a storage area network, all workstations are connected to all storage, so in principle, tape and network copies of data could be eliminated. There is still the problem of coordinating the workstation s access to data, so that one workstation does not begin to edit an object before the predecessor workstation is finished with it. The right side of Figure 7 illustrates how a storage area network can combine with SNPoint Foundation Suite and SNPoint Foundation Suite H to solve this problem. The storage area network provides universal interconnection of computers and data. SNPoint Foundation Suite and SNPoint Foundation Suite H coordinate access to data, so that, for example, video objects are passed between stations only when appropriate (i.e., not in use by any other workstation). This saves time and eliminates a significant source of errors. Page 10 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

13 ackup pplication: lock-level Incremental Database ackups Perhaps the most onerous administrative task in enterprise IT organizations today is backup. The more a business relies on information, the less tolerable data loss becomes. n effective backup and restore strategy is an operational necessity. s fast as backup technology improves, however, increasing application demands, larger databases, and more complex system topologies make it more difficult again. No matter how fast the backup technology, two fundamental backup problems remain: Obtaining consistent point-in-time backups of large file systems or databases without blocking application access for the entire image copy time required to copy the image acking up large file systems or databases without disrupting operational client traffic or server I/O SNPoint Foundation Suite and SNPoint Foundation Suite H make it easy to perform frequent backups of critical systems without interfering with the production system performance. Used in conjunction with VERITS Netackup, SNPoint Foundation Suite and SNPoint Foundation Suite H allow off-host backups from a different server in the cluster, accessing the same shared data. dministrators can create point-in-time snapshots of critical data, including Oracle databases, then use a separate server running Netackup to perform backups based on this snapshot. This offers very low-impact backups, allowing more frequent backups for better data protection. New VERITS Volume Management and File System Technology for Cluster Environments Page 11

14 III. VERITS Cluster Volume Resilient Raw Storage for Clusters VERITS Volume VERITS Volume enhances online storage availability and performance by aggregating disks or hardware-based RID arrays into failure-tolerant, high-performance, flexible logical volumes. Volume Data mapping and presentation Plex Plex Failure tolerance and data mapping Failure tolerance and data mapping Disks Disks Figure 8: VERITS Volume volume architecture Volume implements the three-tier storage object architecture illustrated in Figure 8. lock ranges on physical disks are aggregated into plexes. Each plex has failure tolerance (e.g., RID) and data mapping (e.g., striping) properties. Plexes are further aggregated into volumes that are used by file systems, database managers and applications as if they were disks. Volume Type Disk Failure Tolerance Performance pplication in Enterprise IT (data availability) (relative to single disk) n-way mirrored Very high High read performance, Small (single-disk) (sometimes called RID 1) comparable write critical files performance Striped and n-way mirrored Very high Very high read performance, Large (multi-disk) critical files high write performance Striped with parity (RID 5) High High read performance, low write performance Important read-mostly data Striped without parity Lower than single disk High read and write Easily replaceable but (sometimes called RID 0) performance performance-critical data Concatenated Lower than single disk Comparable read and Easily replaceable, not (capacity aggregated) write performance performance-critical data Table 1: Volume volume types Page 12 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

15 Table 1 lists the types of volumes that Volume supports. For failure tolerance, a major focus in cluster computing, both n-way mirrored volumes and RID volumes are supported. Mirrored volumes offer the greatest protection against disk failure. Moreover, Volume supports mirrored volumes with three or more identical copies of data on separate disks. Three-way mirroring is useful when a frozen image of critical operational data is required for backup or for testing new applications. One copy of a three-way mirrored volume can be disconnected for backup or testing while live applications continue to run with the failure-tolerant data on the other two. Volume can also stripe data across several disks or disk plexes to optimize performance by balancing I/O load. Striping can be used alone, or in conjunction with mirroring or parity RID for failure-tolerant high-performance volumes. Volume also introduces new flexibility in storage capacity management. Volume volumes can be expanded while they are online, enabling a system administrator to respond to unplanned storage needs with little or no application disruption. VERITS Volume has been improving data availability, performance and management flexibility in thousands of enterprise IT systems for several years. The basic Volume design is predicated on a single point of control for each volume; each volume is managed and used by a single host computer. This model, illustrated in Figure 9, has been used successfully in VERITS FirstWatch and VERITS Cluster shared-nothing clusters. Cluster VERITS Volume Either server or server may own the volume, but not both Cluster VERITS Volume VERITS Volume failure-tolerant volume Figure 9: Volume volumes in a VERITS Cluster cluster The failure tolerant volume in Figure 9 may be accessed by one server at any instant. If the application or server using the volume fails, the cluster server transfers ownership of the volume to the server to which the application will fail over. New VERITS Volume Management and File System Technology for Cluster Environments Page 13

16 VERITS Cluster Volume While VERITS Volume has proven to be extraordinarily useful in practice, its single point of control model does not support shared-data clustering (Figure 4, page 5). VERITS has therefore developed the Cluster Volume, based on VERITS Volume, to provide robust raw storage for VERITS Cluster or other (e.g., Oracle Parallel ) shared-data clusters. Cluster Volume is a separately licensed product that adds the following features to Volume : Simultaneous access to volumes from multiple servers Clusterwide logical device naming Consistent logical view of volume state from all servers Volume management from any server in the cluster Volumes that remain accessible by surviving servers after server failures pplication failover without volume failover Like Volume, Cluster Volume is capable of managing both physical disks and the virtual disks exported by hardware RID array subsystems. Cluster Volume rchitectural Concepts Like Volume, the Cluster Volume organizes disks into disk groups. Each volume is allocated from disks in one disk group. Cluster Volume disk groups, and therefore volumes, are either private or cluster-shareable. Even if they are physically connected to the entire cluster, disks in a private disk group may only be accessed by a single server. In Figure 10, disk groups,, C and D are necessarily private because each is connected to a single server. Each server must have a root disk group that is private, whether or not the disk(s) in it are physically connected to more than one server. Private disk group C C VERITS Volume Private disk group VERITS Volume Private disk group S 2 VERITS Volume Private disk group D VERITS Volume Private disk group D Cluster-shareable disk group S 1 Figure 10: Private and cluster-shareable disk groups Page 14 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

17 Cluster-shareable disk groups may be accessed by multiple servers simultaneously. They must be physically connected to all servers in the cluster. In Figure 10, disk group S1 could be a cluster-shareable disk group. Disk group S2 could not be cluster shareable because it is only accessible by servers and. Disk group S2 could be designated as a private disk group owned by either server or server. Cluster Volume Manger supports simple (single-disk), spanned (concatenated), mirrored, striped, striped mirrored and mirrored striped volumes. In later releases, support for parity RID volumes is planned. Cluster-Shareable Volume and Client Nodes s with VERITS Volume, a system administrator uses the vxdg utility to designate that a disk group is private or cluster-shareable. Private disk groups are functionally equivalent Volume disk groups on single-host systems. When any cluster server starts up, a master node (server) must initially form the cluster. Cluster nodes running Cluster Volume should start the Cluster Volume volume reconfiguration daemon during their startup processes. This daemon automatically imports cluster shareable disk groups as it discovers them, making their volumes accessible to the node. The first node to import a cluster shareable disk group becomes the group s master node. s additional servers join the cluster, importing cluster shareable disk groups, they become client users of the disk group. ll servers in a cluster with Cluster Volume import all cluster shareable disk groups at startup. cluster shareable disk group s master node performs all metadata changes for the disk group. Metadata changes include, for example, creating volumes using storage capacity within the disk group, resizing volumes, removing mirror copies from mirrored volumes, and so forth. Whenever disk group metadata is changed, the master uses distributed transaction techniques to synchronize all nodes so that changes take effect simultaneously throughout the cluster. Cluster Volume Feature Overview The Cluster Volume 3.2 release supports the sharing of volumes among up to 16 servers (i.e., 16-node clusters). Cluster Volume is aware of RID controllers that can access disk arrays on more than one path (I/O bus), and can switch access paths for all cluster nodes simultaneously. Rolling software upgrades are supported, allowing a Cluster Volume installation to be upgraded by taking one cluster node at a time out of service. Clustered Volume Semantics Cluster Volume is, as is name implies, a cluster volume manager. It exports volumes with disk-like semantics for use by cluster file systems and other system applications, such as distributed database managers. Cluster Volume synchronizes metadata updates to provide a consistent clusterwide view of volumes in a cluster-shareable disk group. The volumes themselves, however, with respect to application read and write commands behave like disks. Like disks, volumes execute host read and write commands indiscriminately. If a server can physically access a disk, it can issue any valid read or write command to the disk; likewise with a volume. This behavior is appropriate in single-server systems, where each disk or volume is mounted by a single file system or database instance that coordinates all access to it. In a cluster, however, multiple nodes may be reading and writing a disk at the same time. Without coordination among the nodes, data updates could inadvertently overwrite each other, or a read executed during another node s write to an overlapping block range could return an inconsistent in flight image of data. Cluster Volume reproduces this disk semantic. The volume manager does not guarantee that different cluster nodes user data reads and writes will not interfere with each other. Such guarantees must be provided by the file system or database manager using the volume. Cluster Volume is designed for use with cluster-aware data managers such as VERITS Cluster File System. This data manager has distributed lock managers that coordinate access to shared-data objects, such as files or disk blocks, across the cluster. Cluster File System uses VERITS Global Lock to coordinate data access. New VERITS Volume Management and File System Technology for Cluster Environments Page 15

18 Cluster Volume and System Failures Cluster Volume uses VERITS Volume technology to mirror data across two or more disks to provide disk failure tolerance. With a mirrored volume, Volume performs each application write on corresponding blocks of each of the volume s disks. It is possible for a server to fail with writes to mirrored volumes in progress. This can lead to inconsistent volume contents with some copies updated and others not. Guaranteeing consistency of failure-tolerant volume contents after a crash of the control point (the server running the volume manager) is a well-known problem in volume management technology with known solutions. For example, a mirrored volume might be made consistent after a crash by designating one disk as the master, and copying its entire contents to other disks mirrored with it. Solutions like this are time-consuming, however, and reduce data availability and with it application availability. VERITS products are well known for their ability to restore volume and file system consistency quickly after system crashes. For mirrored volumes, dirty region logs (DRLs) indicate which block ranges ( regions ) of the volume have been updated recently, and which therefore may possibly be inconsistent after a system crash. dirty region log entry is written before any disk of a mirrored volume is updated. When all mirrors of a volume have been updated, the region is marked as no longer at risk. To minimize overhead, the log is not written until another pending update requires it. When Cluster Volume recovers from a node crash, it reads its dirty region log and recopies data in mirrored volume regions flagged as at risk. Since only a tiny fraction of a volume s regions are likely to be at risk at any instant, recovery time is much shorter than with brute force copying seconds rather than minutes or hours for very large volumes. Page 16 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

19 IV. VERITS Cluster File System Enabling the Shared-Data Cluster Model File System for Clusters VERITS Cluster Volume makes robust logical volumes accessible to data managers and raw device applications throughout a cluster. VERITS Cluster File System enables several clustered servers to mount and use a file system simultaneously as if all applications using the file system were running on the same server. Figure 11 illustrates Cluster File System in a cluster. Cluster private interconnect Private root file system VERITS SPFS Global Lock Cluster Volume y default, Global Lock uses SN interconnect to coordinate file system actions VERITS SPFS Global Lock Cluster Volume Private root file system Write file X.DT Write file Y.DT Cluster Volume volume containing SNPoint Foundation Suite file system Figure 11: VERITS Cluster File System layered on Cluster Volume New VERITS Volume Management and File System Technology for Cluster Environments Page 17

20 Cluster File System Properties VERITS Cluster File System is based on the single-host VERITS File System. The maturity and rich feature set of VERITS File System have made it the file system of choice in enterprise UNIX environments. Features that make VERITS File System excel in enterprise applications include: Extent-based space management that concisely maps files up to a terabyte in size Fast recovery from most system crashes using a self-clearing intent log to track recent file system metadata updates Online administration capability that allows file systems to be extended and defragmented while they are in use Quick I/O features that allow aware database managers to bypass kernel locking by treating files as raw partitions, and enable 32-bit applications to take advantage of a system cache larger than 4 gigabytes With VERITS Cluster File System, all of these features are available to clusters of servers. Cluster File System also takes advantage of the cluster environment to provide: Clusterwide freezing of file system state, which allows operations that require a consistent on-disk image of a file system, (e.g., removal of a copy from a mirrored volume for backup or testing), to be performed in a cluster environment oth clusterwide and local file system mounting, allowing administrators to share data among cluster nodes or not as application requirements dictate Rolling upgrade of Cluster File System itself, so that it can be upgraded node by node, allowing the cluster as a whole to operate throughout the upgrade process If a server using a Cluster File System file system fails, its applications can be failed over to a surviving server, but in many cases the file system need not be restarted since it is still running. This eliminates most of the application data recovery time, usually the major time-consuming element when applications are restarted after a system crash. VERITS Cluster File System enefits Perhaps the greatest beneficiary of the SNPoint Foundation Suite and SNPoint Foundation Suite H is the system administrator. With Cluster Volume and Cluster File System, many large-system administration tasks that result from hardware limitations are simplified or eliminated: Cluster Volume can create and manage terabyte-size volumes, so partitioning file systems to fit within disk capacity limitations is rarely necessary. Cluster File System can support file systems with up to one terabyte of capacity, so only the very largest data farms need to be partitioned due to file system addressing limitations. Since all servers in a cluster have access to Cluster File System cluster-shareable file systems, maintaining consistent reference data or application images and libraries across multiple servers is automatic. ll cluster nodes can work from the same reference data and images. Moreover, the storage space required for multiple identical copies of applications and reference data in a shared-nothing cluster is unnecessary when all servers are working from the same data and images. Since all files can be accessed by all servers, applications can be allocated to servers to balance load or meet other operational requirements. Similarly, failover is more flexible since it is not constrained by data accessibility. ecause each Cluster File System file system can be mastered on any cluster node, the file system recovery portion of failover time in an n-node cluster can be reduced by a factor of n by distributing the mastership of file systems across cluster nodes. Page 18 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

21 Enterprise RID subsystems can be justified more easily and used more effectively, because all of their storage capacity can be mounted by all servers, and reallocated to meet changing business needs through management operations rather than hardware reconfigurations. The larger volumes with wider striping enabled by file system shareability improve application I/O load balancing. Not only is the I/O load of each server spread across more I/O resources, but with Cluster File System shared file systems, the loads of all servers are balanced across all I/O resources. With fewer volumes and file systems, locating individual objects is easier for administrators, users, application developers and managers, and backup. Extending clusters by adding servers becomes easier, because each new server s data storage configuration is mostly predefined. New servers simply adopt the existing clusterwide volume and file system configuration. The single system-image administrative model of Cluster File System simplifies administration by making all file system management operations (except resizing) independent of the location from which they are invoked (although actual metadata operations are carried out by the current Cluster File System master node). dministrators do not find themselves in the awkward situation of having server capacity available and data to process, but no way to bring the two together. With Cluster File System cluster-shareable file systems, all data is accessible by all servers, with single-server file system semantics. With Cluster File System, the Quick I/O feature that makes file-based databases outperform raw partition-based databases is available to applications running in a cluster. new optional feature in SNPoint Foundation Suite and SNPoint Foundation Suite H called VERITS FlashSnap extends VERITS File System with Storage Checkpoint technology, enabling a system administrator to perform operations such as data analysis, backup, testing and reporting without impacting the production environment. Cluster File System and pplications Virtually all applications should benefit from Cluster File System. Conventional, cluster-unaware applications benefit from being able to run and access their data from anywhere in a cluster. In multi-application clusters, overall I/O performance should improve due to better load balancing if Cluster File System file systems are built on larger volumes. Once Cluster File System is installed, these benefits are automatic. No tuning or other administrative action is necessary. Many applications are partitionable, that is, they consist of multiple concurrent execution threads that could run on different servers if they had a way to coordinate their data accesses. Cluster File System provides this coordination. Such applications can be made cluster-aware, allowing their instances to cooperate to balance client and data access load, and thereby scale beyond the capacity of any single server. In such applications, Cluster File System does not balance I/O load per se, but by providing shared-data access, it enables application-level load balancing across cluster nodes. VERITS Cluster File System rchitecture -Client File System Design VERITS Cluster File System uses a master-client architecture to manage file system metadata on shared volumes. The first server to mount each Cluster File System file system becomes its master; all other nodes in the cluster become clients. pplications access the user data in files directly from the server on which they are running, as Figure 12 indicates. Cluster File System metadata, however, is only updated by the file system s master node (the first node to mount the file system). The Cluster File System master node is responsible for making all metadata updates, and for maintaining the file system s metadata update intent log. If client servers need to update file system metadata (e.g., allocate new files or delete old ones), they do so by communicating their requests to the master, which performs the actual updates and responds to the requesting server. (The SN connecting the cluster servers to their storage is the preferred interconnect for this communication, as well as for internal Cluster Volume and Cluster File System communications.) This design guarantees the consistency of file system metadata and of the intent log used to recover from system crashes. New VERITS Volume Management and File System Technology for Cluster Environments Page 19

22 Private root file system VERITS SPFS Global Lock CVM Metadata updates Create file Z.DT VERITS SPFS Global Lock CVM Private root file system Create file Z.DT Write file X.DT Write file Y.DT Cluster Volume volume containing SNPoint Foundation Suite file system Figure 12: VERITS Cluster File System server-client architecture Cluster File System Failure Tolerance If a Cluster File System master server fails, the surviving cluster nodes elect a new master. The new master reads the file system s intent log and completes any metadata updates that had been in process at the time of failure. pplication I/O stalls during recovery, which typically takes a few seconds. When the file system is again consistent, it is mounted automatically by the new master and by clients, and application processing resumes. Note that the full VERITS Cluster component, only found in SNPoint Foundation Suite H, is required for automatic application failover. Since Cluster File System client nodes do not update file system metadata directly, failure of a Cluster File System client node does not necessitate any metadata repair. Cluster File System recovery from client node failure is therefore typically perceptibly faster than recovery from master node failure. Page 20 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H

23 Cluster File System and the VERITS Global Lock Cluster File System relies on VERITS Global Lock to reproduce UNIX single-host file system behaviors in clusters. The most important is write behavior. UNIX file system writes appear to be atomic. This means that when an application writes data to a file, any subsequent application read to the same file area retrieves the new data, even if it has been cached by the file system and not yet written to disk. pplications never retrieve stale data, or partial results from a prior write. In a single-host system, the file system is a central point of control and knowledge about what data is in cache. In a cluster, each node with Cluster File System cluster-shareable file systems mounted has its own cache. In order to reproduce singlehost write semantics, the caches must be kept coherent each must instantly reflect any application updates to cached data, no matter which cluster node they come from. efore it can read or write a range of blocks in a file, Cluster File System uses Global Lock to lock the file so that no other node in the cluster can update it simultaneously, nor read it before the update is complete. Global Lock locks an object by first electing a master node for the lock. Global Lock sends a lock request to the master (possibly the node making the request), which responds by granting the lock or by attempting to revoke it if it already exists elsewhere in the cluster. When a lock is finally granted, Cluster File System performs the requested I/O, and eventually releases the lock when data has been written to disk. From an application standpoint, Global Lock is as transparent as the locking mechanisms of a local file system. pplications need not be Global Lock -aware. pplications that operate on the same data at the same time must have their own locking mechanisms for the higher-level data objects that they manage. For example, a cluster running a distributed database management system against a database stored in Cluster File System container files would lock database rows to control read and write access by all of the cooperating database instances. (Oracle s Portable Lock performs this function for Oracle Parallel databases.) Cluster File System becomes aware of cluster state transitions (nodes leaving or joining the cluster) through the Global tomic roadcast protocol (see Figure 13). When a server fails, or when a new server joins the cluster, Cluster File System remasters (elects a new master for) all existing locks. When all locks that existed before the cluster transition have been remastered, application processing resumes. From an application standpoint, I/O requests stall during the cluster transition and while locks are being remastered, but otherwise the transition is transparent. New VERITS Volume Management and File System Technology for Cluster Environments Page 21

24 Cluster File System and VERITS Cluster Protocols Figure 13 shows the major functional components that interface with VERITS Cluster File System. The role of the VERITS Global Lock in controlling access to files from different cluster nodes has already been discussed. VERITS SPFS Global Lock CVM = VERITS SPFS Global Lock CVM Cluster File System dmin tools Global Lock (cache coherency management) Global tomic roadcast (membership) Low-Level Transport (messaging) I/O (to disk or through Cluster Volume ) Cluster Volume volume containing SNPoint Foundation Suite file system Figure 13: VERITS Cluster File System integrated components Of great importance to Cluster File System are the two Cluster protocols it uses for determining membership and inter-node communication, the Global tomic roadcast (G) protocol and the Low Latency Transport (LLT) protocol. G and LLT are VERITS Cluster -specific protocols that can be implemented directly on a fibre channel transport (for minimal overhead and latency) or on Ethernet data link protocol. G is a broadcast protocol, in the sense that it guarantees that messages broadcast throughout a cluster are received reliably (i.e., are acknowledged), and are received in the same order by all nodes. The main application of G is to provide a membership service, both for the cluster as a whole, and for groups of applications running it, such as Cluster File System instances. The G membership service provides orderly startup and shutdown. LLT, as the name implies, is optimized for fast delivery of the large numbers of small messages that characterize distributed locking traffic over the simple network topologies that typically characterize clusters. oth G and LLT are designed to run over redundant data links connecting all of the servers in a cluster. Cluster requires redundant cluster communication links to minimize the possibility of cluster partitioning due to failure of a single communication link. Page 22 VERITS SNPoint Foundation Suite and SNPoint Foundation Suite H