Addressing Data Management and IT Infrastructure Challenges in a SharePoint Environment. By Michael Noel

Transcription

1 Addressing Data Management and IT Infrastructure Challenges in a SharePoint Environment By Michael Noel

2 Contents Data Management with SharePoint and Its Challenges...2 Addressing Infrastructure Sprawl and Scalability...2 Understanding the Need for Multiple Farms and Servers...3 Scaling a SharePoint Environment...3 Using Server Virtualization to Address Server Sprawl...4 Providing High Availability and Disaster Tolerance for SharePoint...5 Clustering and Mirroring SharePoint s Data Tier...5 Providing Load Balancing Options for the Web Tier...6 Addressing Service Application Availability...6 Understanding Backup and Restore Requirements for SharePoint...7 Addressing the Content Growth Explosion...8 Content Database and Site Collection Structure Limitations...8 Understanding EBS and RBS BLOB Storage Options...9 Providing Storage Solutions for Managing Data...9 Upgrade and Migration Data Management Challenges...9 Data Management Challenges...10 Rearchitecting the Data Tier...10 Introduction: Data Management with SharePoint and Its Challenges In recent years, the success of Microsoft s SharePoint products and technologies has led to an explosion in data management requirements for many organizations. IT environments struggle to manage the rapid growth of SharePoint, while at the same time ensure that their existing environment will scale to fit their growing needs. SharePoint has a few native controls to help manage data, but organizations that don t implement plans for managing their data growth can become overwhelmed with the sheer volume of data created within a SharePoint environment. In addition, if proper thought is not put into server design, storage considerations, content security, and disaster tolerance, small issues soon turn into larger ones. Organizations need a comprehensive approach to SharePoint design that takes into account all the data management challenges a SharePoint environment creates and that gives administrators control over their infrastructure. This whitepaper addresses the most common types of data management challenges SharePoint administrators face and discusses different solutions organizations have taken to address those challenges. The paper incorporates information obtained from a survey of more than 350 SharePoint administrators, architects, and business decision makers that focused on understanding what the data management challenges are and how they re currently being addressed. Addressing Infrastructure Sprawl and Scalability In a production environment, there are few simple SharePoint deployments. SharePoint was designed to be scalable and distributed, allowing for data to be easily distributed and allowing the environment to grow. It s this scalability, however, that can lead to infrastructure sprawl, a challenge that was identified as a key concern by 29 percent of all respondents to the survey. 2 Data Management with SharePoint and Its Challenges Sponsored by NetApp

3 Understanding the Need for Multiple Farms and Servers A SharePoint farm is a self-contained set of SharePoint servers that forms the largest logical unit that you can design. Each individual farm may be composed of one or several servers that are part of the farm. A single SharePoint server can only be a member of a single farm, with the exception of SQL Server database servers, which can hold databases from multiple farms. There are multiple reasons why organizations may need to deploy multiple farms within their organization, including, but not limited to, the following: Geographic Restrictions A single SharePoint farm does not lend itself well to communicating with other farm members across slower network links such as WAN connections. Farm members need high bandwidth and low latency connections. Subsequently, organizations typically either decide to deploy centralized SharePoint farms and have clients access those farms across the WAN, or they deploy multiple farms in different locations, contributing to infrastructure sprawl. Security Boundaries While there are ways of establishing multi-tenancy within a SharePoint farm, the ultimate security boundary within SharePoint is the farm itself. So, if different sets of administrators need to access a farm, or if some of the farm content is highly sensitive, separate farms may be created for security purposes. Dedicated Services Farm Multiple farms can consume Service Applications that are hosted on other SharePoint farms. For example, several farms within an organization can use the search service application, the user profile sync service, and the managed metadata store on a single centralized farm that provides those services to all farms within an organization. This is particularly true with larger distributed organizations. Development and/or Test Farms Having a place to test service patches, updates, and new features is critical for any production environment, because it greatly reduces the risk associated with updating SharePoint. Even the smallest SharePoint environments should ensure that they have test and/or development farms for this reason. Staging Farms Some environments choose to have staging farms deployed, which serve as a location for SharePoint content to be reviewed before it s published into production. This is particularly the case in web content management scenarios such as those involving published SharePoint web sites on the Internet. Disaster Recovery Farms Depending on the disaster recovery option chosen, it may include the creation of a standby Disaster Recovery farm in a remote location to provide for SharePoint functionality in the event of an outage. For these reasons, and as a general reflection of natural growth within SharePoint, it is very common for administrators to become challenged with the growth and sprawl of a SharePoint environment. Scaling a SharePoint Environment SharePoint 2010 was designed to be highly scalable, and to allow organizations to take small, single-server SharePoint environments and massively scale them to hundreds of thousands of users. It does this by providing the following strategies that can be implemented when the existing environment becomes overcommitted: Distributing Databases SharePoint allows for databases to be distributed across multiple SQL Servers, allowing the data tier to scale very easily. Farm databases such as the search databases can reside on their own SQL Server, for example, or data in the content databases can be split among multiple content databases and stored on multiple servers. Splitting Roles onto Dedicated Servers A SharePoint farm consists of one or more servers that each have a designated role within that farm, such as the web role and the various Service Application roles. Simply by breaking a role off from a server and moving it to a new server, overcommitted servers can have their load reduced, allowing administrators to scale the environment. Adding servers to the farm Adding additional servers to an existing SharePoint farm has the effect of load balancing the role across the various role servers. Adding extra web role servers to a farm, for example, Sponsored by NetApp Data Management with SharePoint and Its Challenges 3

4 allows them to become additional servers within a load balanced pool, offloading some of the web traffic from the other servers in that pool. Using Server Virtualization to Address Server Sprawl Server virtualization technologies are a powerful weapon in the fight against server sprawl. Indeed, today, just over 50 percent of the survey respondents indicated that they re currently using server virtualization for at least part of their SharePoint environment, and many more are considering deploying it as well. Virtualization has advantages that go beyond the basic tenants of consolidation and optimization of the hardware; it also gives SharePoint architects the design flexibility to be able to deploy farm servers that are dedicated to a specific role, but without the need to deploy additional physical hardware. To illustrate, in Figure 1 the SharePoint architect has taken advantage of server virtualization to deploy multiple farms on a small number of systems. In this case, the SharePoint role systems are deployed across two hosts, and the databases are distributed between a pair of physical SQL Servers. The virtualization allows this particular model the flexibility of having multiple farms and multiple dedicated servers running on a much smaller number of physical systems; in this case, four. This type of design flexibility gives virtualization an edge over traditional design techniques, and allows for better control over infrastructure sprawl. Microsoft supports SharePoint in a virtualized environment, per the guidelines established as part of the Server Virtualization Validation Program (SVVP), outlined at As a caveat, while supported, multiple factors must be taken into account before virtualizing SharePoint, including, but not limited to, the following: Consider Passthrough/RDM Devices The best performance is gained on a virtual guest by allocating what Microsoft refers to as Passthrough devices (also known as Figure 1: Sample Server Virtualization Model Raw Device Mapping, or RDM devices) to the individual guests. This is a concept of taking a physical disk or NIC and directly mapping it to the virtual guest, rather than using a virtualized disk or NIC. Consider the advantages and disadvantages of this approach, however, because using Passthrough devices limits the ability to use virtual host failover solutions. Scale out Individual Tiers An equivalent virtual machine will not get the same performance as a physical one, and you may need to scale out an individual tier more quickly to get better performance. Fortunately, virtualization allows you to do this quickly and easily simply be deploying more virtual guests to offload capacity from other farm members. Don t Push the Limits Using memory and process overcommit technologies is one of the biggest source of performance issues in a SharePoint farm. Be very cautious when using them. At the same time, highly consider dedicated virtual hosts for your SharePoint environment as well. Closely Watch Performance Disk I/O, processor, and memory must be closely watched to ensure that the virtual guests are not being overwhelmed. This involves collecting performance counters from the hosts and guests and watching for greater than normal disk read/write times (Higher 4 Data Management with SharePoint and Its Challenges Sponsored by NetApp

5 than 15ms), high processor utilization (Higher than 90 percent), high network bandwidth (Higher than 60 percent), and low memory conditions. If these exist, consider scaling out the tier of servers or add performance with better storage or passthrough/rdm devices. Consider Quick Provisioning Options Virtualization management software allows for the creation of server templates that you can use to quickly add SharePoint servers into a farm through a scripted process. Consider creating SharePoint template servers that already have SharePoint installed, but not configured. After provisioning these servers, they can be quickly added into an existing farm or be used to set up a new farm using PowerShell provisioning techniques. Providing High Availability and Disaster Tolerance for SharePoint In the survey of SharePoint admins and decision makers, more than 74 percent of respondents indicated that SharePoint is a mission-critical or very important tool for their organization. SharePoint has in many cases become as important to organizations as or phones, and is considered mission critical. It s subsequently critical for most organizations to ensure that the environment is made both highly available and disaster tolerant. Shared Storage Clustering The most traditional high availability option for SQL servers involves the creation of a shared storage cluster: multiple nodes have access to the physical storage where the databases are stored and can take over the processing of those databases in the event of an outage of one or more of the nodes. Clustering is available in either Standard or Enterprise editions of SQL Server (though the standard edition only supports a two-node cluster.) Some third-party solutions also provide for Geo-clustering capabilities, which allow for nodes of a cluster to be located across a WAN. SQL Database Mirroring SQL Database Mirroring is newer than clustering, having been introduced with SQL Server 2005 and improved in SQL Server 2008 and SQL Server 2008 R2. Mirroring combines both high availability and disaster recovery options into a single solution. It works by creating a mirrored copy of the SharePoint databases on a completely Clustering and Mirroring SharePoint s Data Tier One of the most visible and critical tiers in a SharePoint architecture is the data tier. Indeed, all the SharePoint content is stored in this tier; recovering it is the most important requirement for disaster recovery of SharePoint content. SQL Server, which runs the SharePoint data tier, includes multiple options that provide for a combination of disaster tolerance and high availability. These include the following: Figure 2: SQL Server High Availability Mirroring for SharePoint Sponsored by NetApp Data Management with SharePoint and Its Challenges 5

6 different SharePoint server. When used together with a third witness server, such as Figure 2 shows, Database Mirroring allows for the SharePoint databases to be able to failover to the mirror server. Note, however, that asynchronous mirroring (typically used in WAN failover scenarios) can t mirror certain system databases such as the config DB; only synchronous mirroring is supported in these scenarios. Providing Load Balancing Options for the Web Tier The web tier of SharePoint, run by Windows Server Internet Information Services (IIS) running on web role servers, can be easily load balanced to provide for high availability. Load balancing takes the form of two supported options: Hardware-Based Load Balancing Many organizations take advantage of dedicated hardware-based load balancers, which take the web traffic destined for SharePoint and distribute it between multiple SharePoint web servers. The only caveat with using hardware-based load balancers is that the session affinity of the connection must be set to single in normal operations, a concept also referred to as sticky sessions. Because SharePoint is a session-based application, this means that a single client s traffic must be routed through a single server, unless that server fails. Not configuring the load balancer this way can lead to an interruption of the user s session with the server. Windows Network Load Balancing Microsoft s Windows Server Network Load Balancing (NLB) is a fully supported solution as well. NLB runs as a service on the web servers, and is used to balance the traffic sent to SharePoint. Windows Server has two types of NLB:, Unicast, which requires dedicated network cards for the heartbeat traffic, and multicast, which requires special router configuration. Addressing Service Application Availability SharePoint is divided into three tiers, which Figure 3 shows. This includes the user-facing web tier, the data tier, where all of the information is stored, and a third tier. The third tier, new in SharePoint 2010, is the tier where the various SharePoint 2010 Service Applications operate. The Service Application tier is an extensible layer comprised of multiple Service Applications that perform various workloads throughout a farm. Service Applications include common services such as search, but also include other more complex functionality such as the Managed Metadata Service, which provides a location for centralized configuration of metadata and content types, ensuring consistency across the types of data within an organization. The number and type of service applications provided varies depending on the version of SharePoint utilized. Table 1 illustrates the native SharePoint 2010 Service Applications, what they re used for, and which versions of SharePoint they re included in. Because the Service Application tier is extensible, third-party Service Applications can also be written and Microsoft may expand this list in the future. Typically, addressing high availability within Service Applications involves simply adding an additional server to a farm and configuring that farm to run the Service Application as well. Because most service applications store their data within custom Service Application databases, this makes it easier to allow for multiple servers at the Service Application tier to exist, offloading some of the workload and providing for high availability in the event of an outage of one of the servers. Figure 3: SharePoint Architecture Layers 6 Data Management with SharePoint and Its Challenges Sponsored by NetApp

7 Table 1: Service Application Matrix Service Application Description SharePoint Foundation 2010 SharePoint Server 2010 Standard SharePoint Server 2010 Enterprise Access Services Business Data Connectivity service Excel Services Application Lets users view, edit, and interact with Access 2010 databases in a Web browser. Gives access to line-of-business data systems. Lets users view and interact withexcel 2010 files in a Web browser. Managed Metadata service Manages taxonomy hierarchies, keywords and social tagging infrastructure, and publish content types across site collections. PerformancePoint Service Application Search service Secure Store service State service Usage and Health Data Collection service User Profile service Visio Graphics Service Provides the capabilities of PerformancePoint. Crawls content, produces index partitions, and serves search queries. Provides single sign-on authentication to access multiple applications or services. Provides temporary storage of user session data for SharePoint Server components. Collects farm wide usage and health data, and provides the ability to view various usage and health reports. Adds support for My Site Web sites, profile pages, social tagging and other social computing features. Lets users view and refresh published Visio 2010 diagrams in a Web browser. Web Analytics service Provides Web service interfaces. Word Automation Services Microsoft SharePoint Foundation Subscription Settings Service Performs automated bulk document conversions. Provides multi-tenant functionality for service applications. Tracks subscription IDs and settings for services that are deployed in partitioned mode. Deployed through Windows PowerShell only. Some Service Applications, such as Search, require more complex customization, but can still be configured to be highly available. Indeed, SharePoint 2010 now allows for redundant search indexes to be created, something that wasn t possible with SharePoint Understanding Backup and Restore Requirements for SharePoint SharePoint 2010 is extremely capable and scalable, but those capabilities come with a price in complexity. Understanding all the moving parts in the various tiers of SharePoint is a complex task, and is a source of significant anxiety among SharePoint administrators, with data protection issues being a major concern for 33 percent of the survey respondents. Microsoft provides for several native backup and restore options for SharePoint 2010, including the following: Backup and Restore from the SharePoint Central Administration Tool This is the most obvious method of backing up SharePoint, directly from the web-based Central Admin tool. New features in SharePoint 2010 include status bars that Sponsored by NetApp Data Management with SharePoint and Its Challenges 7

8 Figure 4: Sample Best Practice Content Database and Site Collection Layout indicate the progress of the backup and the ability to get more granular, backing up at the Site Collection level. PowerShell Scripted Backup Processes SharePoint backups can now be automated using PowerShell, which kicks off the same process as the SharePoint Central Admin Tool does, though from a command-line. It provides for the same level of granularity. SQL Server Database Backups using SQL Server Maintenance Plans The most common native backup approach to SharePoint involves backing up the SharePoint databases using a SQL Server Maintenance Plan. However, this approach doesn t back up the search index, or any custom configuration made to individual SharePoint servers. The most important factor to understand when planning for SharePoint backup and restore is that the native tools do not fully address all the concerns most organizations have in terms of SharePoint backup and restore. For example, no built-in tools provide for individual item-level restore from SharePoint backups. Backup and restore times, as well as the amount of storage required for multiple backups, can also be greatly optimized. Most enterprise organizations look to third-party solutions for help in this area. Addressing the Content Growth Explosion The rapid growth of content within SharePoint environments is a significant area of concern for many organizations. In the survey, 29 percent of respondents indicated that rapidly growing server and 8 Data Management with SharePoint and Its Challenges Sponsored by NetApp

9 storage requirements presented a significant challenge. In addition, more than 86 percent indicated that refreshing their server and storage infrastructure was important. Tools and methods to deal with this explosion in content growth are required. Content Database and Site Collection Structure Limitations How SharePoint is logically structured contributes to challenges around data management. Specifically, the fact that SharePoint Site Collections can only exist in a single content database can severely restrict the scalability of the data tier, particularly for those organizations that don t install multiple site collections and content databases when initially creating their SharePoint farm. Ideally, site collections and content databases should be deployed so that they re distributed, as Figure 4 shows. In this model, the SharePoint environment is used for document management, and each department within the organization is given its own Site Collection, each installed into a separate content database. With this approach, the data is distributed across the environment from the beginning, and the total size of individual content databases can be brought under control. As a general guideline, Microsoft recommends not allowing individual content databases to grow larger than 200GB. Using this strategy can help avoid individual databases from approaching this size. Understanding EBS and RBS BLOB Storage Options For larger organizations or those with significantly higher data storage requirements, Microsoft provides a supported option within the SharePoint data tier that allows for documents stored in SharePoint to be physically stored outside the content database. This concept works by taking the Binary Large OBjects (BLOBs), which include documents, photos, and anything other than the metadata and properties of the document, and storing it in a separate storage location. Because 80 percent to 90 percent of the space taken up in a SharePoint content database is comprised of BLOBs, this can have the effect of drastically reducing the size of the content databases. The original, SharePoint-specific BLOB storage option was the External BLOB Storage (EBS) option. EBS was created for SharePoint 2007, and is still supported in SharePoint Microsoft has indicated, however, that EBS won t be supported in the next version of SharePoint. The more recent BLOB storage option is a SQLspecific technology known as Remote BLOB Storage (RBS.) RBS operates at the SQL layer, so SharePoint is not even aware that the content is separated. This has performance enhancements, but the one limiting factor of RBS over EBS is that it must be enabled for all BLOBs in a single database, whereas EBS can be more granular. When using EBS or RBS, you must take into account factors such as how it changes backup and restore requirements, and the fact that it changes the way that other SQL features such as database mirroring and SQL Transparent Data Encryption (TDE) operate. That said, proper use of an RBS/EBS solution, which typically takes the form of a third-party option that uses the API, can reduce the overall data management challenges with SharePoint Providing Storage Solutions for Managing Data The majority of SharePoint performance issues are directly related to issues at the data tier, so it s incredibly important to ensure that the data tier is properly designed to allow for high levels of disk I/O. Typically, this means housing the SharePoint databases on a fast Storage Area Network (SAN) or equivalent storage infrastructure. Indeed, 74 percent of participants in the survey indicated that they are using SAN disk with their SharePoint environment. For planning SharePoint disk I/O requirements, Microsoft gives the equation of 0.75 IOPS per GB of SharePoint content, although this can vary depending on the nature of the SharePoint environment, the number of users, and other factors. Work directly with your storage vendor to implement the proper amount of disk performance for SharePoint and to determine the appropriate storage architecture. Take into account the different workloads your business will be implementing through SharePoint s feature set (eg., Workflow, Document Automation, ECM). The various workloads will help drive the discussion around anticipated storage requirements for the Sponsored by NetApp Data Management with SharePoint and Its Challenges 9

10 business in terms of amount, growth, and data type for performance and cost optimization. Also consider availability requirements for the various content and business processes being relied upon SharePoint. Upgrade and Migration Data Management Challenges Even a year after SharePoint 2010 was released, only 22 percent of survey respondents indicated that they re already using SharePoint 2010 in production. However, 38 percent indicated that they plan on migrating to this new version within the next six months. This makes upgrade and migration a major topic of concern for many organizations. Indeed, because the architecture of the back-end SharePoint environment has changed so much, this poses challenges for these organizations that must plan their migration strategy carefully to avoid running into the types of data management challenges listed in this document. Data Management Challenges The biggest issue from a data management perspective that most organizations face when migrating to SharePoint 2010 is how to architect the data tier to allow for both environments to co-exist during the migration process. In addition, changes to the taxonomy between SharePoint 2007 and SharePoint 2010 also create challenges when migrating content between systems. Microsoft provides two migration options natively, an in-place upgrade and a separate approach known as the database attach migration process. The in-place upgrade has significant disadvantages, namely risk and inflexibility, so it s not commonly used. This means that the database attach process, which involves building a brand new SharePoint 2010 environment and then migrating to that environment database by database, ends up being the preferred option. A third option is to rely on third-party migration tools, which also assume the creation of a new parallel SharePoint 2010 environment. Because the database attach processes and thirdparty migration tool processes require a separate newly created SharePoint farm, organizations must plan on new storage for the SharePoint 2010 farm, because the existing legacy SharePoint storage is still utilized throughout the process. If there is a large amount of data, this can lead to some serious challenges in balancing the storage requirements during the migration. Rearchitecting the Data Tier Because of these migration challenges, organizations often restructure and rearchitect their storage environment during the migration process. This has the side effect of allowing for Greenfield deployment for the new farm, giving administrators the ability to design around best practices and to build for future performance and scalability. It also provides for an opportunity to implement the other best practices discussed, such as SQL Server Database mirroring, geo-clustering, or third-party storage solutions that provide for better HA and DR of the new target environment. As evident in the survey results and as discussed in this white paper, SharePoint administrators and architects face significant data management challenges that can lead to performance issues, infrastructure sprawl, and other serious problems. Putting the proper thought into architecture of the three tiers of SharePoint 2010, including web, Service Application, and data, can help minimize the effects of these challenges and provide for a robust and high performance SharePoint 2010 environment that can scale to the needs of your organization. Michael Noel is an MVP for SharePoint Server and an MCSE+I. He has been involved in the computer industry for nearly two decades, and has significant real-world experience helping organizations realize business value from Information Technology. Currently a partner at Convergent Computing in the San Francisco Bay Area, Michael s writings and worldwide public speaking experience leverage his real-world expertise designing, deploying, and administering IT infrastructure for his clients. To learn more about NetApp solutions for SharePoint Server environments, go to microsoft-sharepoint-server/ 10 Data Management with SharePoint and Its Challenges Sponsored by NetApp