Integrated Data Management:

Transcription

1 Integrated Data Management: An Innovative Approach to Improved IT Productivity and Flexibility VS Joshi, Ravi Chalaka, Network Appliance, April 2006, TR-3467 Executive Summary Typical IT organizations have formed separate but highly interdependent groups to manage applications, servers, data, and storage. The administrators in each group are highly specialized and focused on the efficient management of their individual areas, but many tasks require collaboration between functional areas for effective execution. The absence of linkage between these areas has resulted in operational inefficiencies and rapid escalation in storage management costs. Integrated data management (IDM) is a new approach to simplifying data management and improving IT productivity by linking application data requirements to storage management processes. The IDM approach combines process automation with role-based access security to deliver significant increases in IT management efficiency and storage resource utilization. In the current environment of changing business conditions, the IDM approach increases flexibility of the business by enabling owners of the data to take quick action on the data. This is done within the confines set by the storage administrator.

2 Table of Contents 1. Introduction Traditional Storage Management: Managing the Containers A New Approach Is Needed Integrated Data Management Integration and Automation Datasets Policies Maximum Productivity for IT Operations IDM: An Innovative Approach to Simplifying Data Management... 8 Conclusion

3 1. Introduction In a typical IT organization, separate groups manage applications, servers, data, and storage, and each group focuses on the efficient management of its particular area. In this environment, traditional storage management processes have evolved to satisfy the needs of the storage administrator to discover, monitor, and control storage devices and other elements of the storage infrastructure. These processes have had little linkage to those used by system, application, and database administrators, who as a group might be called data administrators, in their efforts to establish and sustain a high quality of service (QOS) level for business applications and processes. To meet their objectives, this latter group concentrates on the control and management of data and would prefer to do so without dealing with the details and intricacies of storage infrastructure management. To put it another way, traditional approaches to storage and storage resource management may be thought of as dealing with storage devices, capacity, and connections, while data management concerns itself with data movement, protection, and recovery. This difference in focus has led to a natural divergence in the development of management processes that typically focus narrowly on satisfying the needs of either the storage administrators or the data administrators with little linkage or correlation between the activities of these groups. In effect, this has led to operational inefficiency and a high degree of interdependence among the different classes of administrators. Integrated data management (IDM) is an innovative approach to simplifying data management that encompasses both the management of storage devices and the data that resides on those devices. Using an IDM approach, storage administrators operate more efficiently and with minimal interruptions by automating routine storage management processes and by linking those processes to specific application requirements. This is accomplished without sacrificing control over the storage environment by defining appropriate, reusable policies that support different QOS requirements for each application. By creating linkages between application requirements and storage management processes in a controlled environment, IDM addresses the needs of system, application, and database administrators to control their data in a language that they understand, without the need for extensive storage management skills. Because the data owners can perform certain data management tasks by themselves, their ability to respond to changing business conditions is enhanced, thus increasing the flexibility of the entire business. The use of process automation, role-based access, and policy-based management enables business-centric control of data and reduces the interdependencies between storage and data administrators to deliver dramatic productivity and flexibility gains. NetApp is continuing its tradition of simplifying data management by assuming a leadership role in defining integrated data management. 2. Traditional Storage Management: Managing the Containers The management of storage began as a hardware-oriented activity, with relatively few and simple software tools. While these tools were adequate to handle early storage environments, which often were supplied by a single vendor and supported only one application and operating environment, the situation today is much different. The continuing explosive growth of enterprise data represents ever-increasing demands for storage capacity. At the same time, storage environments have become much more complex, with distributed applications and storage, pervasive networking, multiple operating environments, and different classes of tiered networked storage being deployed. These changes have spurred the development of storage management approaches geared toward managing the increased capacity and complexity. These new strategies have generally retained a storage-centric approach to the problem, with little or no linkage to application requirements. Trends in storage management using traditional methods have succeeded in masking the complexity of the problem rather than simplifying the solution. They give IT administrators limited ability to be self-sufficient in responding to changing data requirements, leaving them little alternative to managing storage as a collection of containers and connections. 3

4 Only by understanding data in an application context can intelligent storage management choices be made to deliver the required level of performance, availability, and data protection. From the database or application administrator's point of view, the data or contents are of the highest level of concern, while the details of the storage configuration (providing the configuration supports the desired service level) are of little importance. 3. A New Approach Is Needed In traditional IT organizations application, database, system, and storage administrators each focus narrowly on only a part of the data/storage management problem. Each has different and distinct areas of responsibility and accountability. As a result, end-to-end data management depends upon manual communication between the data administrator and storage administrator and manual mapping of data to storage and upon the storage administrator to perform the actual data movement and copy operations. This is a disruptive and potentially error-prone process that results in critical errors and lost productivity. One way to understand how the traditional approach can contribute to inefficiency is to examine how the needs of the different administrators help shape the storage and data management workflow. Table 1 highlights some of these needs. Application administrators Enhance data protection/access/retention service levels Support for dynamic data growth Reduced dependence on other admins Data administrators Support for data classification and compliance initiatives Simplified management of data protection and disaster recovery Easy data migration and storage consolidation Server administrators Simplified storage provisioning without learning storage skills Server-independent management of storage Automated LUN management/attachment Storage administrators Freedom from interruption Consolidated management of storage infrastructure Ability to focus on high-value activities (performance, utilization, RPO, RTO) Table 1) The important things to notice in Table 1 are that each of the administrators requires a different set of storage services and that, with the exception of the storage administrator, the central focus of their efforts is not directly on the management of storage. For example, as database sizes increase, application and database administrators can be expected to require additional high-performance storage allocated and provisioned to support their availability and QOS requirements. They will also need to create periodic dataconsistent snapshot copies and possibly copies on secondary storage to support data protection, disaster recovery, and compliance. Finally, in the event of failure, the proper files, tables, and logs must be located and used for system recovery. In the traditional approach, each of these situations triggers a series of dialogues and requests among application, database, systems, and storage administrators. The typical storage provisioning or recovery 4

5 process can be both inefficient and error-prone, since it involves dozens of steps that must be performed correctly and in the proper sequence by different people in different parts of the IT organization. Figure 1 illustrates a traditional IT organization in which application and server administrators call upon the storage administrator to satisfy even the most routine requests for storage services. This affords the storage administrator a high degree of control over the storage environment but subjects the storage administrator to frequent interruptions. Dependence on the storage administrator as a help desk, troubleshooter, and recovery facilitator detracts from the ability to concentrate on higher-value storage management activities. To address these challenges, IT organizations have little choice but to assign more people to keep the mission-critical environment from being underserved. Ultimately, both storage utilization and management efficiency tend to decline dramatically as the scale and complexity of the storage environment increase. Figure 1) In one common approach to dealing with these inefficiencies, interdisciplinary teams are formed to manage critical application areas such as ERP, messaging, and business intelligence. Each of these teams draws upon the resources and specialized knowledge of the storage, server, database, and application administrators and collectively manages the infrastructure associated with a particular application. The advantage of this approach is that it encourages increased communication and sharing of localized expertise and has the potential to improve cross-organization efficiencies. Its downside is that it still requires personal involvement from multiple administrative layers even for routine activities. This often results in inefficient utilization of operational resources across the groups. Because functional capabilities and 5

6 responsibilities remain distributed across the administrative layers of the IT organization in this model, application service levels are hard to manage, and process automation is difficult. In many organizations, operational inefficiencies have pushed the cost of managing storage to a disproportionate and unacceptably high fraction of the overall storage budget. The underlying cause of this inefficiency can be traced back to a mismatch between the different interests and needs of the various administrative groups and the processes and software capabilities available to them. To overcome the shortcomings in these areas, a fundamentally different approach to storage management is needed. 4. Integrated Data Management 4.1 Integration and Automation Integrated data management is a new approach that takes an application-centric and data-centric view of storage management and allows the controlled automation of essential storage management activities. The strategy of IDM is to integrate many of the key functions of data management with applications and to allow each administrative layer to become more self-sufficient. IDM satisfies the needs of the storage administrator by reducing the frequency of outside interruptions without sacrificing management control over the storage environment. It addresses the needs of server, application, and database administrators to control their data and, within limits, to manage their own storage requirements without the need for extensive storage management skills. This win-win combination allows each of the administrative groups to function more autonomously and to work together more efficiently and helps reduce the overall cost of managing both storage and data. One way to understand how moving from traditional methods to an IDM approach can improve efficiency is to think of it as analogous to the banking industry deploying ATM machines. When an ATM is used, customers no longer need to stand in line and wait for someone else to help them deposit or withdraw their money or to verify balances. Instead, the process has been automated to give customers secure self-service access to their own funds and account information 24 hours a day. Throughout the process the bank remains in complete control of how much money can be taken out and to which accounts the customer has access. These controls are defined by preestablished policies and are enforced by automated processes. This system is better for the bank because there aren t lines of people waiting for service, so bank employees can be more efficient and can concentrate on higher value activities. It s also better for the customers, who no longer have to wait for service and have access to their funds and account information whenever they want. In much the same way an ATM provides banking customers with self-service access to funds and information, the IDM approach can provide application and system administrators with secure, controlled, self-service access to storage capacity and services. This gives them more flexibility in their choices and allows them to respond more quickly to changing conditions. To understand how this takes place, it is first necessary to understand some underlying concepts that are central in enabling IDM. 4.2 Datasets Datasets separate the notion of data from the physical storage on which it resides and group together related data elements for a particular use or application. A dataset consists of one or more copies of data associated with a particular application or data use. For example, a dataset for a business-critical database application might include two or more synchronized replicas of the active database residing on high-performance primary storage. These may be collocated or housed in different locations to support high application availability and disaster recovery. It would also be likely to include some number of secondary copies such as backups, point-in-time snapshot copies, transaction logs, and other files and information necessary to recover the database following data loss or corruption. Finally, the dataset might include long-term archival copies and data held to meet regulatory or 6

7 business practice compliance requirements. Additional datasets consisting of different groups of related data elements may be defined to support specific activities (for example, backup) for the same application. Datasets for different uses and applications typically vary in structure to accommodate different business priorities. They may contain different numbers of copies or be allocated to different classes of storage. These decisions need to be made based on the quality of service requirements of the application and on the capabilities, cost, and availability of different classes of storage and may involve input from a number of different parts of the IT and business management organizations. In an IDM context the utility of a dataset is that it represents a predictable, reusable, application-specific structure that makes it much easier to manage and provision storage for an existing application or a new instance of the application. This predictability, plus the flexibility to create multiple datasets for different uses, makes it possible to combine datasets with policies to automate many storage and data management activities. 4.3 Policies A policy defines the business logic regarding how data is to be managed (that is, protected, copied, moved, provisioned, or accessed) and defines a set of actions to be applied to a dataset to perform a particular operation or when certain conditions are met. Policies enable automated data management with distributed control. Any number of policies may be defined to accomplish different tasks associated with a particular dataset. For example, one policy may specify that primary data storage must be provisioned only from the highest performance storage system class and that such storage be allocated only in replicated pairs. Another policy may determine the time interval between creation of snapshot copies and specify the location or class of storage where they should be stored. A third policy may describe how data is to be selected for movement to long-term archive; a fourth policy is related to how long it is to be retained, a fifth policy about how it is to be disposed of, and so on. Many applications are supplied with a set of predefined policies or policy templates for common data management tasks. These can be used directly or can be modified easily by the storage administrator to create policies that support an integrated data management approach. Just as dataset definition requires collaboration, establishing effective policies is likely to require input from different parts of the organization. This dialogue between the groups allows requests stated in terms of application service objectives, for example, to be translated by the storage administrator into a series of repeatable storage management and data movement activities that taken together constitute a policy. In the future, similar application storage service requests can be met simply by invoking this same policy. 4.4 Maximum Productivity for IT Operations A major objective of integrated data management is to create the most productive IT operations environment possible. The use of policies and datasets is central to the IDM approach and makes it possible to safely and repeatably automate a great many storage and data management processes to satisfy higher-level service requests. Policy-based automation gives application, database, and storage administrators ATM-like self-service access to storage capacity and services. In the IDM model, the application administrator can get storage provisioned as needed, right away, without having to interrupt the storage administrator or wait for a response, providing the request is within the limits of the policies that have been set. This reduces the turnaround time to satisfy these requests and leads to greater efficiency and productivity. Another advantage to the use of policies is that they are created with input from the storage administrator and can take full advantage of the most advanced storage system features for example, virtualization, thin provisioning, and instant snapshot copies that might otherwise be overlooked by the application, database, 7

8 or system administrator. Figure 2 shows how the storage administrator is relieved of interruptions and operational workflows are improved when datasets and policies are combined in an IDM environment. Figure 2) It's important to emphasize that this automation does not take the storage administrator out of the loop, nor does it reduce the administrator s ability to control the storage environment. It just changes the nature of the job and allows the storage administrator to be more productive. Instead of focusing on a ceaseless stream of interruptions, the administrator can concentrate on higher value storage management functions such as creating effective policies, improving storage utilization, and managing tiered storage. Just as the bank retained complete control over access to funds and account information in the ATM example, the storage administrator continues to control the storage environment by defining appropriate automation policies. 5. IDM: An Innovative Approach to Simplifying Data Management Integrated data management is intended to provide a holistic approach to managing data by combining the functions and benefits of data, storage, and storage resource management. Two primary technology elements that allow the IDM model to work are effective storage virtualization and policy-based automation. Without these it would be difficult or impossible to achieve the goals of integrated data management. For IDM to be widely adopted, its benefits must appeal to a wide range of information technology professionals by promising to simplify their daily activities. The IDM approach would support the most commonly used enterprise operating systems and would be integrated with enterprise databases and applications, including Oracle, SAP, Microsoft Exchange, and Microsoft SQL Server. It would leverage familiar concepts, make use of management consoles and other tools already in place, and provide for future growth and expansion by providing open interfaces and making use of standards-based technology. Finally, it would provide role-based security and access control to a simple, uniform, and highly usable human interface. 8

9 Figure 3) IDM represents a new paradigm of storage management in which requirements are set as data priorities linked to the needs of different applications and data uses. Such a solution is illustrated in Figure 3. Virtualization and automation allow storage administrators to easily and dynamically manage massive storage pools and to create controlled, self-service policies for use by application and system administrators. This means many application requests for storage capacity and services may be satisfied automatically while storage administrators maintain control over the storage environment. Under policy control, data can be replicated, stored, and migrated among different classes of storage to satisfy the needs of different virtualized datasets for data protection, business continuance, disaster recovery, archiving, and compliance. Conclusion Traditional approaches have left a gap between the management of data and the management of storage. This has resulted in inefficient operations, with considerable duplication of effort and with frequent interruptions to the activities of highly interdependent administrative groups. Integrated data management is an emerging, holistic approach to managing both data and storage more efficiently. In this approach, policy-based automation is used to deliver self-service administrative access to many storage and data management activities without sacrificing control over the storage environment. With IDM, data administrators can maximize the benefit from storage investments by using policies to take full advantage of the most advanced storage system features. IDM also elevates interactions between data and storage administrators to a higher functional level, reducing the frequency of interruptions and making both groups more productive. Finally, IDM shortens the time needed to react to changing conditions, increasing the flexibility and responsiveness of the data management process. 9

10 Figure 4) Many of the elements of a successful integrated data management approach can be deployed today and can deliver dramatic improvements in IT productivity and storage resource utilization. NetApp is continuing its efforts to simplify data management by assuming a leadership role in defining integrated data management Network Appliance, Inc. All rights reserved. Specifications subject to change without notice. NetApp and the Network Appliance logo are registered trademarks and Network Appliance and Snapshot are trademarks of Network Appliance, Inc. in the U.S. and other countries. Microsoft and Windows are registered trademarks of Microsoft Corporation. Oracle is a registered trademark of Oracle Corporation. UNIX is a registered trademark of The Open Group. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.